Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34684

Jcloud - Unable to create build nodes in Openstack cloud

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: jclouds-plugin
    • Labels:
    • Environment:
      Jenkins Version: 1.580.1
      jclouds plugin: 2.8.1-1
      Openstack Versions: Icehouse - nova, keystone, neutron; Kilo - glance, cinder
    • Similar Issues:

      Description

      In the last few days, our Jenkins master has lost the ability to communicate with our Openstack installation. When jenkins tries to create a slave, or reconnect to existing slaves after a Jenkins restart, there is an error:

      ...
      May 09, 2016 9:17:33 AM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: Error parsing input
      com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 930
      	at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:181)
      ...
      

      Complete log file is attached as well as a sample response from Openstack.

      Our Openstack engineer has examine the cloud configuration, and has not been able to identify any issues there.

      Restarting the service & the host did not clear up the issue. We did not change any Jenkins config, and this was working normally previously.

        Attachments

          Activity

          Hide
          felfert Fritz Elfert added a comment -

          Obviously, something in the openstack response has changed (some single value was expected, but an array value was returned). Similar exceptions have been noticed before, when some openstack components had been updated.
          To find out the exact reason, please upload the json response again, but NOT beautified but as a single line. Otherwise it is impossible to determine what element started at column 930 originally.

          Show
          felfert Fritz Elfert added a comment - Obviously, something in the openstack response has changed (some single value was expected, but an array value was returned). Similar exceptions have been noticed before, when some openstack components had been updated. To find out the exact reason, please upload the json response again, but NOT beautified but as a single line . Otherwise it is impossible to determine what element started at column 930 originally.
          Hide
          davegoodine dave goodine added a comment -

          Fritz Elfert - here is the json:

          {"access": {"token": {"issued_at": "2016-05-09T20:31:38.918294", "expires": "2016-05-10T20:31:38Z", "id": "96471d3bdaf245de950a5e8c98dd96cf", "tenant": {"description": "", "enabled": true, "id": "be351b66fff74a25a1e060f101a1889a", "name": "cibuild"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://localhost:8888", "region": "regionOne", "internalURL": "http://localhost:888", "id": "7dbfb467567b400d81a3eeb018e0a41a", "publicURL": "http://localhost:8888"}], "endpoints_links": [], "type": "rating", "name": "CloudKitty"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a", "id": "33dc4970833847db8280ace6fa7dff32", "publicURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:9696", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:9696", "id": "5e68688c7d1247d39fa0b5535f323840", "publicURL": "http://controller-n02.openstack.drillinginfo.com:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a", "id": "866d4b882b044387955d1809fc3248df", "publicURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:9292", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:9292", "id": "4b7000e012a94082bfcc3e1745dbe21e", "publicURL": "http://controller-n01.openstack.drillinginfo.com:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8777", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8777", "id": "3f5714f2514247be8947285c7b951dbb", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8777"}], "endpoints_links": [], "type": "metering", "name": "ceilometer"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1", "id": "55e2b168ac454e369f4207b5a757c4be", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1"}], "endpoints_links": [], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"adminURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a", "id": "cfa7beecf0064c9181bd4d27aebf31e8", "publicURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a", "id": "46b092a9a68a4bec89d9bdbf56257bdd", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:35357/v2.0", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:5000/v2.0", "id": "5706a782cffa4a8ebfa4b3f884771cc2", "publicURL": "http://controller-n02.openstack.drillinginfo.com:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "cibuild-openstack", "roles_links": [], "id": "cibuild-openstack", "roles": [{"name": "_member_"}], "name": "cibuild-openstack"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab"]}}}
          
          Show
          davegoodine dave goodine added a comment - Fritz Elfert - here is the json: { "access" : { "token" : { "issued_at" : "2016-05-09T20:31:38.918294" , "expires" : "2016-05-10T20:31:38Z" , "id" : "96471d3bdaf245de950a5e8c98dd96cf" , "tenant" : { "description" : "", " enabled ": true , " id ": " be351b66fff74a25a1e060f101a1889a ", " name ": " cibuild "}}, " serviceCatalog ": [{" endpoints ": [{" adminURL ": " http: //localhost:8888 ", " region ": " regionOne ", " internalURL ": " http://localhost:888 ", " id ": " 7dbfb467567b400d81a3eeb018e0a41a ", " publicURL ": " http://localhost:8888 "}], " endpoints_links ": [], " type ": " rating ", " name ": " CloudKitty "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a ", " id ": " 33dc4970833847db8280ace6fa7dff32 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " compute ", " name ": " nova "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:9696 ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:9696 ", " id ": " 5e68688c7d1247d39fa0b5535f323840 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:9696 "}], " endpoints_links ": [], " type ": " network ", " name ": " neutron "}, {" endpoints ": [{" adminURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a ", " id ": " 866d4b882b044387955d1809fc3248df ", " publicURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " volumev2 ", " name ": " cinderv2 "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:9292 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:9292 ", " id ": " 4b7000e012a94082bfcc3e1745dbe21e ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:9292 "}], " endpoints_links ": [], " type ": " image ", " name ": " glance "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8777 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8777 ", " id ": " 3f5714f2514247be8947285c7b951dbb ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8777 "}], " endpoints_links ": [], " type ": " metering ", " name ": " ceilometer "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 ", " id ": " 55e2b168ac454e369f4207b5a757c4be ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 "}], " endpoints_links ": [], " type ": " cloudformation ", " name ": " heat-cfn "}, {" endpoints ": [{" adminURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a ", " id ": " cfa7beecf0064c9181bd4d27aebf31e8 ", " publicURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " volume ", " name ": " cinder "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a ", " id ": " 46b092a9a68a4bec89d9bdbf56257bdd ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " orchestration ", " name ": " heat "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:35357/v2.0 ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:5000/v2.0 ", " id ": " 5706a782cffa4a8ebfa4b3f884771cc2 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:5000/v2.0 "}], " endpoints_links ": [], " type ": " identity ", " name ": " keystone "}], " user ": {" username ": " cibuild-openstack ", " roles_links ": [], " id ": " cibuild-openstack ", " roles ": [{" name ": " _member_ "}], " name ": " cibuild-openstack "}, " metadata ": {" is_admin ": 0, " roles ": [" 9fe2ff9ee4384b1894a90878d3e92bab"]}}}
          Hide
          marisasawat Marisa Sawatphadungkij added a comment -

          Fritz Elfert Hello, do you have any suggestions?

          Show
          marisasawat Marisa Sawatphadungkij added a comment - Fritz Elfert Hello, do you have any suggestions?
          Hide
          davegoodine dave goodine added a comment -

          Deleting an image with bad metadata resolved this issue.

          Show
          davegoodine dave goodine added a comment - Deleting an image with bad metadata resolved this issue.

            People

            • Assignee:
              felfert Fritz Elfert
              Reporter:
              davegoodine dave goodine
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: