Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44809

Multibranch pipeline: StackOverflowException in BranchJobProperty#hasPermission

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      Jenkins 2.64 on Linux, latest version of all plugins (pipeline, declarative, blue ocean etc)
    • Similar Issues:

      Description

      A declarative multibranch pipeline causes StackOverflowException in
      org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      when clicking on the pipeline in the (old/not-blue-ocean) Jenkins UI to show the list of branches - see attached screenshot "JenkinsUI-StackOverflowException.png" and stacktrace "SOE in Jenkins UI" below.

      The SOE occurred for the first time after making this change to Jenkinsfile:

      diff --git a/Jenkinsfile b/Jenkinsfile
      index f4e5053..b7f412f 100644
      --- a/Jenkinsfile
      +++ b/Jenkinsfile
      @@ -1,7 +1,9 @@
       @Library('my-shared-library') _
       
       pipeline {
      -    agent any
      +    agent {
      +        label 'test'
      +    }
           
           options {
               timestamps()
      

      The previous commit applied these changes (basically copied from gitlab-plugin README), which may also be relevant:

      diff --git a/Jenkinsfile b/Jenkinsfile
      index 5f2ecb2..f4e5053 100644
      --- a/Jenkinsfile
      +++ b/Jenkinsfile
      @@ -5,7 +5,7 @@ pipeline {
           
           options {
               timestamps()
      -        //[$class: 'GitLabConnectionProperty', gitLabConnection: 'my-gitlab-connection'] // Needed? How to do in declarative pipelines?
      +        gitLabConnection('my-gitlab-connection')
               //skipDefaultCheckout() // if checkout should be done manually later one
               buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '185', numToKeepStr: ''))
               
      @@ -13,6 +13,10 @@ pipeline {
               //   "Additional behaviours" > "Checkout to specific local branch"
           }
           
      +    triggers {
      +            gitlab(triggerOnPush: true, triggerOnMergeRequest: true, branchFilterType: 'All')
      +    }
      +    
           tools {
               jdk 'jdk8'
               maven 'maven3.3.x'
      

      The first SOE stacktrace occurring in jenkins.log is also pasted below as "First and second SOE in jenkins.log", it shows where the SOE originates from.

      Since then, it is not possible anymore to open this pipeline, always getting the attached screenshot.

      Also the SOE now prints to jenkins.log even when looking at other jobs/views (both freestyle and other pipelines). However those other jobs work fine and dont seem to be affected by the SOE printed to jenkins.log. I can attach some of these other SOE stacktraces if required.

      SOE in Jenkins UI
      java.lang.StackOverflowError
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      	at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      [...]
      
      First and second SOE in jenkins.log
      Jun 10, 2017 10:29:48 PM com.dabsquared.gitlabjenkins.webhook.GitLabWebHook getDynamic
      INFO: WebHook called with url: /project/<pipeline-name>
      Jun 10, 2017 10:29:48 PM org.jruby.javasupport.JavaMethod invokeDirectWithExceptionHandling
      INFO: gitlab web hook triggered for
         - repo url: git@git.<git-url>.git
         - branch: MigrateJenkinsfileToDeclarativePipelineSyntax
         - with payload:
      {
      [... http post payload from gitlab ...]
      }
      Jun 10, 2017 10:29:48 PM org.jruby.javasupport.JavaMethod invokeDirectWithExceptionHandling
      INFO: matching projects:
         - <some-jenkins-job-name>
      Jun 10, 2017 10:29:48 PM org.jruby.javasupport.JavaMethod invokeDirectWithExceptionHandling
      WARNING: no project references the given repo url and commit branch
      Jun 10, 2017 10:29:54 PM jenkins.branch.MultiBranchProject$BranchIndexing run
      INFO: <build-pipeline-name> #20170610.222948 branch indexing action completed: SUCCESS in 5.2 sec
      Jun 10, 2017 10:30:03 PM org.eclipse.jetty.server.handler.ContextHandler$Context log
      WARNING: Error while serving https://ci.company.com/view/<view-name>/job/<build-pipeline-name>/job/MigrateJenkinsfileToDeclarativePipelineSyntax/21/logText/progressiveHtml
      java.lang.reflect.InvocationTargetException
              at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347)
              at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
              at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
              at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
              at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
              at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
              at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
              at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
              at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
              at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
              at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
              at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
              at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
              at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
              at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
              at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
              at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
              at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.Server.handle(Server.java:564)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
              at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
              at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
              at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
              at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.StackOverflowError
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      
      [...]
      
      Jun 10, 2017 10:30:12 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      INFO: <build-pipeline-name>/MigrateJenkinsfileToDeclarativePipelineSyntax #21 completed: SUCCESS
      Jun 10, 2017 10:30:13 PM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
      SEVERE: A thread (Thread-5/66) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
      java.lang.StackOverflowError
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
              at org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty$1.hasPermission(BranchJobProperty.java:77)
      
      [...]
      

      A few hours before this SOE occurred I upgraded several plugins. However after upgrading and restarting Jenkins, the same pipeline has built successfully several times. Also I tried downgrading all of them again, with no success.

      I also tried to downgrade the following plugins based on feedback from jenkins-users mailinglist, but that did not help either:

      • "Dashboard for Blue Ocean" plugin from 1.0.1 to 1.0.0
      • "External Monitor Job Type Plugin" from 1.7 to 1.6

        Attachments

        1. 2017-06-23_12-21-01.png
          294 kB
          Alwyn Pan
        2. afterUpgradeTo1.1.6_config.xml.gz
          11 kB
          Stefan Thurnherr
        3. image-2017-06-20-11-11-52-044.png
          302 kB
          Anthony Mastrean
        4. image-2017-06-24-10-25-56-507.png
          14 kB
          Alwyn Pan
        5. image-2017-06-24-10-27-18-582.png
          106 kB
          Alwyn Pan
        6. JenkinsUI-StackOverflowException.png
          606 kB
          Stefan Thurnherr

          Issue Links

            Activity

            Hide
            abayer Andrew Bayer added a comment -

            You've upgraded to the latest release of Pipeline: Model Definition? (which is currently 1.1.7) And you ran that script from the Jenkins Script Console but still got that error? Hrm. Well, if you need to, you can manually edit the file containing all those duplicates and remove all those lines, then restart Jenkins?

            Show
            abayer Andrew Bayer added a comment - You've upgraded to the latest release of Pipeline: Model Definition ? (which is currently 1.1.7) And you ran that script from the Jenkins Script Console but still got that error? Hrm. Well, if you need to, you can manually edit the file containing all those duplicates and remove all those lines, then restart Jenkins?
            Hide
            gaieges Evin Callahan added a comment -

            Yessir to all of the above - Pipeline 1.1.7 and Jenkins 2.66.

            Removing all the duplicate entries from a broken job's config.xml files actually does work, so thank you. I noticed that it's with specific jobs (ie one job may break while the other wont).

            However, after fixing that one job, and trying your script again, it fails in the same way. I presume this is because I have the same problem in other jobs / branches. I do have some of those duplicate entries in other config.xml files, so I'll just go in and delete the ones that have the duplicates manually for now.

            Show
            gaieges Evin Callahan added a comment - Yessir to all of the above - Pipeline 1.1.7 and Jenkins 2.66. Removing all the duplicate entries from a broken job's config.xml files actually does work, so thank you . I noticed that it's with specific jobs (ie one job may break while the other wont). However, after fixing that one job, and trying your script again, it fails in the same way. I presume this is because I have the same problem in other jobs / branches. I do have some of those duplicate entries in other config.xml files, so I'll just go in and delete the ones that have the duplicates manually for now.
            Hide
            abayer Andrew Bayer added a comment -

            Oy! As ridiculous as this sounds, you could probably disable security very briefly, run the script, and re-enable it, but that's probably stupid. =)

            Another also fairly stupid option could be something like this:

            cd $JENKINS_HOME/jobs
            find . -name config.xml -exec sed -i .bak '/<org\.jenkinsci\.plugins\.workflow\.multibranch\.BranchJobProperty reference=\"\.\.\/org\.jenkinsci\.plugins\.workflow\.multibranch\.BranchJobProperty\"\/>/d' {} \;
            # Restart Jenkins
            

            Wouldn't deal with any other massively duplicated property, but it would unblock you from running the script/running the jobs, which should suffice. Just a thought/possibility.

            Show
            abayer Andrew Bayer added a comment - Oy! As ridiculous as this sounds, you could probably disable security very briefly, run the script, and re-enable it, but that's probably stupid. =) Another also fairly stupid option could be something like this: cd $JENKINS_HOME/jobs find . -name config.xml -exec sed -i .bak '/<org\.jenkinsci\.plugins\.workflow\.multibranch\.BranchJobProperty reference=\ "\.\.\/org\.jenkinsci\.plugins\.workflow\.multibranch\.BranchJobProperty\" \/>/d' {} \; # Restart Jenkins Wouldn't deal with any other massively duplicated property, but it would unblock you from running the script/running the jobs, which should suffice. Just a thought/possibility.
            Hide
            gaieges Evin Callahan added a comment -

            I actually ended up doing something along the lines of your sed command after finding the biggest offenders via this:

            find ./jobs -name 'config.xml' -exec grep -H BranchJobProperty {} \; | uniq -c | sort -n
            

            And then manually removing the big offenders. Once I did that, I was able to run your script and things are working peachy now!

            Thanks again!

            Show
            gaieges Evin Callahan added a comment - I actually ended up doing something along the lines of your sed command after finding the biggest offenders via this: find ./jobs -name 'config.xml' -exec grep -H BranchJobProperty {} \; | uniq -c | sort -n And then manually removing the big offenders. Once I did that, I was able to run your script and things are working peachy now! Thanks again!
            Hide
            alwynpan Alwyn Pan added a comment -

            Andrew Bayer is that a way to stop / clean these jobs stucked in the history? Thank you.

             

            Show
            alwynpan Alwyn Pan added a comment - Andrew Bayer is that a way to stop / clean these jobs stucked in the history? Thank you.  

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                stefanthurnherr Stefan Thurnherr
              • Votes:
                2 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: