Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47224

Pipeline builds failing/hanging under load: RejectedAccessException: Scripts not permitted to use staticMethod WorkflowScript ___cps___6009

    Details

    • Similar Issues:

      Description

      When we have many pipeline build running simultaneously, we see them starting to hang, never completing or failing (screenshot attached)

      In the logs it shows that it failed with the following error:

      org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod WorkflowScript __cps__6009

      We're not actually calling this method anywhere, so I think it's happening within jenkins workflow framework itself:

      Here's a bigger chunk of the logs of one of the failed builds:

      Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7"
      Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T12:39:06-07:00)
      Maven home: /opt/apache-maven-3.5.0
      Java version: 1.8.0_144, vendor: Oracle Corporation
      Java home: /usr/java/jdk1.8.0_144/jre
      Default locale: en_US, platform encoding: ANSI_X3.4-1968
      OS name: "linux", version: "2.6.32-642.3.1.el6.x86_64", arch: "amd64", family: "unix"
      [jenkins-maven-event-spy] INFO generate /u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7/maven-spy-20170929-114954-962.log ...
      [INFO] Scanning for projects...
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/maven-metadata.xml
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-resources-plugin/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/maven-metadata.xml
      (812 B at 1.8 kB/s)
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-install-plugin/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/maven-metadata.xml
      (623 B at 9.2 kB/s)
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/codehaus/mojo/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
      (13 kB at 133 kB/s)
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
      (20 kB at 122 kB/s)
      [INFO] Downloaded:
      http://repository.mulesoft.org/releases/org/apache/maven/plugins/maven-metadata.xml
      (567 B at 1.5 kB/s)
      [INFO] Downloading:
      http://repository.mulesoft.org/releases/org/codehaus/mojo/versions-maven-plugin/maven-metadata.xml
      [INFO] Downloading:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/versions-maven-plugin/maven-metadata.xml
      [INFO] Downloaded:
      https://repo.maven.apache.org/maven2/org/codehaus/mojo/versions-maven-plugin/maven-metadata.xml
      (742 B at 11 kB/s)
      [INFO]
      [INFO] ------------------------------------------------------------------------
      [INFO] Building Stores WCS Experience API 1.0-SNAPSHOT
      [INFO] ------------------------------------------------------------------------
      [INFO]
      [INFO]

            • maven-clean-plugin:2.5:clean (default-clean) @ stores-wcs-exp-api ---*
              [INFO]
              [INFO]
            • versions-maven-plugin:2.5:lock-snapshots (default-cli) @ stores-wcs-exp-api ---*
              [INFO] ------------------------------------------------------------------------
              [INFO] BUILD SUCCESS
              [INFO] ------------------------------------------------------------------------
              [INFO] Total time: 3.179 s
              [INFO] Finished at: 2017-09-29T11:49:58-07:00
              [INFO] Final Memory: 20M/304M
              [INFO] ------------------------------------------------------------------------
              [jenkins-maven-event-spy] INFO generated /u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3@tmp/withMavene1cf6ac7/maven-spy-20170929-114954-962.log
              [Pipeline] }
              [withMaven] Archive artifact pom.xml under com/petco/api/stores-wcs-exp-api/1.0-SNAPSHOT/stores-wcs-exp-api-1.0-SNAPSHOT.pom
              [withMaven] Scan Tasks for Maven artifact MavenArtifact{com.petco.api:stores-wcs-exp-api::1.0-SNAPSHOT} in source directory src/main/java
              [TASKS] Scanning folder '/u02/jenkins/workspace/-stores-wcs-exp-api_develop-TYYQL4JJXC7KYU7EN62XWRXQCBCGHF7CSKW3A74EH4XDXLJKRPQA@3' for files matching the pattern 'src/main/java/*/' - excludes: null
              [TASKS] Found 2 files to scan for tasks
              [TASKS] Found 0 open tasks.
              [Pipeline] End of Pipeline
              [BFA] Scanning build for known causes...
              [BFA] No failure causes found
              [BFA] Done. 0s
              org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod WorkflowScript __cps__6009
              at org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.StaticWhitelist.rejectStaticMethod(StaticWhitelist.java:189)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onStaticCall(SandboxInterceptor.java:150)
              at org.kohsuke.groovy.sandbox.impl.Checker$2.call(Checker.java:179)
              at org.kohsuke.groovy.sandbox.impl.Checker.checkedStaticCall(Checker.java:183)
              at org.kohsuke.groovy.sandbox.impl.Checker$checkedStaticCall.callStatic(Unknown Source)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222)
              at WorkflowScript.<clinit>(WorkflowScript)
              Caused: java.lang.ExceptionInInitializerError
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
              at java.lang.reflect.Constructor.newInstance(Unknown Source)
              at java.lang.Class.newInstance(Unknown Source)
              at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:434)
              at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
              at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:516)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:613)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:588)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:612)
              at hudson.model.RunMap.retrieve(RunMap.java:225)
              at hudson.model.RunMap.retrieve(RunMap.java:57)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380)
              at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:345)
              at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:362)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.getPreviousBuild(WorkflowRun.java:226)
              at org.jenkinsci.plugins.workflow.job.WorkflowRun.getPreviousBuild(WorkflowRun.java:132)
              at hudson.plugins.analysis.core.BuildHistory.getAction(BuildHistory.java:137)
              at hudson.plugins.analysis.core.BuildHistory.getReferenceAction(BuildHistory.java:123)
              at hudson.plugins.analysis.core.BuildHistory.getReferenceAnnotations(BuildHistory.java:106)
              at hudson.plugins.analysis.core.BuildResult.initialize(BuildResult.java:278)
              at hudson.plugins.analysis.core.BuildResult.<init>(BuildResult.java:222)
              at hudson.plugins.tasks.TasksResult.<init>(TasksResult.java:92)
              at hudson.plugins.tasks.TasksResult.<init>(TasksResult.java:59)
              at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:187)
              at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:68)
              at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:295)
              at org.jenkinsci.plugins.pipeline.maven.reporters.TasksScannerReporter.process(TasksScannerReporter.java:100)
              at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:126)
              at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$Callback.finished(WithMavenStepExecution.java:770)
              at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
              at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
              at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:165)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162)
              at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
              at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
              at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
              at java.util.concurrent.FutureTask.run(Unknown Source)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
              at java.util.concurrent.FutureTask.run(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)
              Finished: FAILURE

        Attachments

          Issue Links

            Activity

            Hide
            abayer Andrew Bayer added a comment - - edited

            Sam Van Oort - yeah, this is related to the reparsing-on-load stuff, i.e. JENKINS-39489. The RejectedAccessException here is technically correct, but it never should have gotten to that point in the first place. Fixing JENKINS-39489 will also fix this, since it's only possible when we're reparsing scripts for a completed build in a context other than actually executing a Pipeline.

            Show
            abayer Andrew Bayer added a comment - - edited Sam Van Oort - yeah, this is related to the reparsing-on-load stuff, i.e. JENKINS-39489 . The RejectedAccessException here is technically correct, but it never should have gotten to that point in the first place. Fixing JENKINS-39489 will also fix this, since it's only possible when we're reparsing scripts for a completed build in a context other than actually executing a Pipeline.
            Hide
            svanoort Sam Van Oort added a comment -

            Andrew Bayer Maybe I'm missing a nuance here, but why are we reparsing the script? Or did you mean to link another JIRA? JENKINS-39489 is mostly concerned with moving where the scripts are persisted to somewhere other than the build itself to reduce the disk hit of loading the build.

            Show
            svanoort Sam Van Oort added a comment - Andrew Bayer Maybe I'm missing a nuance here, but why are we reparsing the script? Or did you mean to link another JIRA? JENKINS-39489 is mostly concerned with moving where the scripts are persisted to somewhere other than the build itself to reduce the disk hit of loading the build.
            Hide
            abayer Andrew Bayer added a comment -

            Ah, right, sorry, JENKINS-45585 - so looks like you already fixed it. =)

            Show
            abayer Andrew Bayer added a comment - Ah, right, sorry, JENKINS-45585 - so looks like you already fixed it. =)
            Hide
            svanoort Sam Van Oort added a comment -

            Ah yeah, the lazy-load of the execution DOES mean we don't have a risk of reparsing the Groovy, excellent. ymenager you should have your fix available in workflow-job 2.18 (you'll want to pick up the latest workflow-cps though due to other fixes to related issues though).

            Show
            svanoort Sam Van Oort added a comment - Ah yeah, the lazy-load of the execution DOES mean we don't have a risk of reparsing the Groovy, excellent. ymenager you should have your fix available in workflow-job 2.18 (you'll want to pick up the latest workflow-cps though due to other fixes to related issues though).
            Hide
            svanoort Sam Van Oort added a comment -

            Not a true duplicate of JENKINS-45585, but the fix to that issue resolves this one.

            Show
            svanoort Sam Van Oort added a comment - Not a true duplicate of JENKINS-45585 , but the fix to that issue resolves this one.

              People

              • Assignee:
                Unassigned
                Reporter:
                ymenager ymenager
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: