Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39282

Jenkins failing to launch cloud agents with OpenStack cloud plugin

    Details

    • Similar Issues:

      Description

      I was using the OpenStack-cloud-plugin to launch new agents on demand when a new pipeline job was executed. It was working perfectly for many days, recently, since a couple days ago launching a new cloud agent fails with this error message:

      Stack trace

      javax.servlet.ServletException: java.lang.NoSuchMethodError: com.google.common.base.Objects$ToStringHelper.omitNullValues()Lcom/google/common/base/Objects$ToStringHelper;
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
      	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
      	at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
      	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NoSuchMethodError: com.google.common.base.Objects$ToStringHelper.omitNullValues()Lcom/google/common/base/Objects$ToStringHelper;
      	at org.openstack4j.openstack.compute.domain.NovaServer.toString(NovaServer.java:276)
      	at jenkins.plugins.openstack.compute.JCloudsSlaveTemplate.provision(JCloudsSlaveTemplate.java:248)
      	at jenkins.plugins.openstack.compute.JCloudsSlaveTemplate.provisionSlave(JCloudsSlaveTemplate.java:170)
      	at jenkins.plugins.openstack.compute.JCloudsCloud.doProvision(JCloudsCloud.java:439)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	... 63 more
      
      If I look into the log of the system this is what I see:
      
      Oct 26, 2016 8:08:34 PM INFO jenkins.plugins.openstack.compute.JCloudsSlaveTemplate provision
      Provisioning new openstack node ci-openstack-agent-small-9767 with options jenkins.plugins.openstack.compute.SlaveOptions@67e1b5c[imageId=ubuntu-server-14.04,hardwareId=2,networkId=fd0e26e9-5705-491e-b8ad-4872880c3d77,userDataId=jenkins.plugins.openstack.compute.UserDataConfig.1470757175885,instanceCap=10,floatingIpPool=<null>,securityGroups=ci-infra-secgrp,availabilityZone=<null>,startTimeout=600000,keyPairName=ci-infra-kp,numExecutors=5,jvmOptions=<null>,fsRoot=/home/ubuntu/,credentialsId=fb4fe359-96e5-49ae-8cbd-cd4737230b71,slaveType=SSH,retentionTime=90]
      Oct 26, 2016 8:08:45 PM WARNING org.eclipse.jetty.util.log.JavaUtilLog warn
      Error while serving http://172.99.106.115/jenkins/cloud/Cloud1/provision
      java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
      	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
      	at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
      	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NoSuchMethodError: com.google.common.base.Objects$ToStringHelper.omitNullValues()Lcom/google/common/base/Objects$ToStringHelper;
      	at org.openstack4j.openstack.compute.domain.NovaServer.toString(NovaServer.java:276)
      	at jenkins.plugins.openstack.compute.JCloudsSlaveTemplate.provision(JCloudsSlaveTemplate.java:248)
      	at jenkins.plugins.openstack.compute.JCloudsSlaveTemplate.provisionSlave(JCloudsSlaveTemplate.java:170)
      	at jenkins.plugins.openstack.compute.JCloudsCloud.doProvision(JCloudsCloud.java:439)
      	... 73 more
      

      Oct 26, 2016 8:11:31 PM INFO hudson.model.AsyncPeriodicWork$1 run
      Started OpenStack slave cleanup
      Oct 26, 2016 8:11:31 PM INFO hudson.model.AsyncPeriodicWork$1 run
      Finished OpenStack slave cleanup. 1 ms
      Oct 26, 2016 8:16:31 PM INFO hudson.model.AsyncPeriodicWork$1 run
      Started OpenStack slave cleanup
      Oct 26, 2016 8:16:31 PM INFO hudson.model.AsyncPeriodicWork$1 run
      Finished OpenStack slave cleanup. 1 ms

        Attachments

          Issue Links

            Activity

            Hide
            olivergondza Oliver Gondža added a comment -

            Downgrading to 2.11 should be safe and resolve the problem as it was introduced in 2.12. Working on a fix.

            Show
            olivergondza Oliver Gondža added a comment - Downgrading to 2.11 should be safe and resolve the problem as it was introduced in 2.12. Working on a fix.
            Hide
            olivergondza Oliver Gondža added a comment -

            The problem is resolved in 2.13 I am releasing right now. Thanks for the report. Unfortunately, this can not be unittested so regression sneaked in 2.12 during a cleanup.

            Show
            olivergondza Oliver Gondža added a comment - The problem is resolved in 2.13 I am releasing right now. Thanks for the report. Unfortunately, this can not be unittested so regression sneaked in 2.12 during a cleanup.
            Hide
            castulo Castulo Martinez added a comment -

            I have confirmed that the problem is no longer happening with 2.13 Thanks!

            Show
            castulo Castulo Martinez added a comment - I have confirmed that the problem is no longer happening with 2.13 Thanks!
            Hide
            olivergondza Oliver Gondža added a comment -

            Cool, thanks!

            Show
            olivergondza Oliver Gondža added a comment - Cool, thanks!

              People

              • Assignee:
                olivergondza Oliver Gondža
                Reporter:
                castulo Castulo Martinez
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: