Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54885

Azure VM Plugin: 1: The storage account name already exists. Use a different name.

    Details

    • Type: Task
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: azure-vm-agents-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.138.1
      Azure VM Agents plugin : 0.7.3
      Jenkins server in a docker container on Ubuntu server, also hosted in MS Azure.

    • Similar Issues:
    • Released As:
      0.7.5

      Description

      We were using  a Storage Account which had issue and we changed our configuration Storage Account Name to use "Create New" this morning.

      It worked fine to start with but later when we changed the "Max Jenkins Agents Limit" from 10 to 50, there was no more agents coming up after first 10 and logs showed the following error:

      "The storage account name already exists. Use a different name. Nov 27, 2018 8:37:35 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision AzureVMCloud: provision: template sshfs has just verified failed"

      We changed the config and used "Storage Account Name" > "Use Existing" and chose one from the available lists.
      Click the "Verify Template"
      and see this error:
      "The following errors occurred while validating the template.
      1: The storage account name already exists. Use a different name."

      Is it trying to create the storage account ? while I am expecting it to just "use existing"
      We have three different cloud configs and I even created a new one, same issue with all.
      I can not say if this is new or we just started to face it, but for some time our autoscaling had been working all fine.

      It is worth mentioning that we made changes this morning since no VM was coming up and there were errors like:
      Failed to create storage account with account name ..
      Status code 429, {"error":{"code":"TooManyRequests","message":"The request is being throttled."}}

        Attachments

          Issue Links

            Activity

            faizan Muhammad Faizan ul haq created issue -
            Hide
            alexb_adv Alex Brown added a comment -

            discovered similar issue, same error message today. apparently been happening since 21/11. 

            Show
            alexb_adv Alex Brown added a comment - discovered similar issue, same error message today. apparently been happening since 21/11. 
            Hide
            psftw Peter Salvatore added a comment -

            We appear to be in the same boat, except we have made no changes to Azure or Jenkins recently.  About six hours ago we started seeing the "The storage account ..." errors and we are up to ~1800 instances.

            Show
            psftw Peter Salvatore added a comment - We appear to be in the same boat, except we have made no changes to Azure or Jenkins recently.  About six hours ago we started seeing the "The storage account ..." errors and we are up to ~1800 instances.
            jieshe Jie Shen made changes -
            Field Original Value New Value
            Assignee Azure DevOps [ azure_devops ] Jie Shen [ jieshe ]
            jieshe Jie Shen made changes -
            Link This issue is duplicated by JENKINS-54899 [ JENKINS-54899 ]
            azure_devops Azure DevOps made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            jackchentoronto Jack Chen added a comment - - edited

            Same here since yesterday, I see a meaningful log:

            Nov 28, 2018 1:01:56 AM com.microsoft.azure.vmagent.AzureVMAgentTemplate$DescriptorImpl doFillExistingStorageAccountNameItems INFO: Cannot list storage account: {0} com.microsoft.azure.CloudException: Status code 400, {"error":{"code":"InvalidApiVersionParameter","message":"The api-version '2016-01-01' is invalid. The supported versions are '2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04'."}}: The api-version '2016-01-01' is invalid. The supported versions are '2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04'. 

             

            Looks like there is apiVersion change in Azure management service and it doesn't support 2016-01-01 anymore.

             

            https://docs.microsoft.com/en-us/azure/templates/microsoft.storage/2016-01-01/storageaccounts

            still list 2016-01-01 as a valid apiVersion.

             

             Actually , manually deployed a storage account and it's working 

            {
                "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
                "contentVersion": "1.0.0.0",
                "resources": [
                    {
                        "name": "sajacktest123",
                        "type": "Microsoft.Storage/storageAccounts",
                        "apiVersion": "2016-01-01",
                        "sku": {
                            "name": "Standard_LRS"
                        },
                        "location": "westus"
                    }
                ]
            } 

             so maybe the problem is not with the apiversion 2016-01-01,

             

            Show
            jackchentoronto Jack Chen added a comment - - edited Same here since yesterday, I see a meaningful log: Nov 28, 2018 1:01:56 AM com.microsoft.azure.vmagent.AzureVMAgentTemplate$DescriptorImpl doFillExistingStorageAccountNameItems INFO: Cannot list storage account: {0} com.microsoft.azure.CloudException: Status code 400, { "error" :{ "code" : "InvalidApiVersionParameter" , "message" : "The api-version '2016-01-01' is invalid. The supported versions are '2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04' ." }}: The api-version '2016-01-01' is invalid. The supported versions are '2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04' .   Looks like there is apiVersion change in Azure management service and it doesn't support 2016-01-01 anymore.   https://docs.microsoft.com/en-us/azure/templates/microsoft.storage/2016-01-01/storageaccounts still list 2016-01-01 as a valid apiVersion.    Actually , manually deployed a storage account and it's working  { "$schema" : "http: //schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#" , "contentVersion" : "1.0.0.0" , "resources" : [ { "name" : "sajacktest123" , "type" : "Microsoft.Storage/storageAccounts" , "apiVersion" : "2016-01-01" , "sku" : { "name" : "Standard_LRS" }, "location" : "westus" } ] }  so maybe the problem is not with the apiversion 2016-01-01,  
            Hide
            jieshe Jie Shen added a comment -

            I have investigated this issue. For now I notice that the API side just misses some information and it causes some exceptions in our codes. I will make a patch to fix this. Sorry for your inconvenience.

            Show
            jieshe Jie Shen added a comment - I have investigated this issue. For now I notice that the API side just misses some information and it causes some exceptions in our codes. I will make a patch to fix this. Sorry for your inconvenience.
            Hide
            jackchentoronto Jack Chen added a comment -

            Yes I see ./src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java file has line

             if (!isAvailable && checkResult.reason().equals(Reason.ACCOUNT_NAME_INVALID))

            and it caused exception, seems the API didn't return reason.  I patched it myself and was able to create a new agent in my test environment. 

            Show
            jackchentoronto Jack Chen added a comment - Yes I see ./src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java file has line if (!isAvailable && checkResult.reason().equals(Reason.ACCOUNT_NAME_INVALID)) and it caused exception, seems the API didn't return reason.  I patched it myself and was able to create a new agent in my test environment. 
            Hide
            jieshe Jie Shen added a comment -

            Jack Chen, Yes the problem is here. I have made a fix and will release a new version after all the tests are passed.

            Show
            jieshe Jie Shen added a comment - Jack Chen , Yes the problem is here. I have made a fix and will release a new version after all the tests are passed.
            Hide
            kkkuba Jakub Michalec added a comment -

            Jie Shen just curious, is Jenkins CI also don't use azure vm's?  I just watch build from PR and it's 'hanging' waiting for VM's

            Show
            kkkuba Jakub Michalec added a comment - Jie Shen just curious, is Jenkins CI also don't use azure vm's?  I just watch build from PR and it's 'hanging' waiting for VM's
            Hide
            jieshe Jie Shen added a comment -

            Jakub Michalec I have no idea about this. I don't have the access to their configuration.

            Show
            jieshe Jie Shen added a comment - Jakub Michalec I have no idea about this. I don't have the access to their configuration.
            Hide
            olblak Olivier Vernin added a comment -

            Jakub Michalec true it is and with exactly the same issue INFRA-1923
            We are affected since yesterday

            Show
            olblak Olivier Vernin added a comment - Jakub Michalec true it is and with exactly the same issue INFRA-1923 We are affected since yesterday
            Hide
            jieshe Jie Shen added a comment -

            The new version 0.7.5 has just been released and should fix this issue. It may take a little time to syn on Jenkins plugin manager. Sorry for all your inconvenience again.

            Show
            jieshe Jie Shen added a comment - The new version 0.7.5 has just been released and should fix this issue. It may take a little time to syn on Jenkins plugin manager. Sorry for all your inconvenience again.
            Hide
            jackchentoronto Jack Chen added a comment -

            Thanks for the quick fix Jie Shen !  Any idea what the root cause is ?  

            Show
            jackchentoronto Jack Chen added a comment - Thanks for the quick fix Jie Shen !  Any idea what the root cause is ?  
            Hide
            jieshe Jie Shen added a comment -

            Jack Chen, it is caused by inconsistent  results from API side. I have created an issue on github repo azure-rest-api-specs . And I think the root cause of this issue may on the service side changes.

            Show
            jieshe Jie Shen added a comment - Jack Chen , it is caused by inconsistent  results from API side. I have created an issue on github repo azure-rest-api-specs  . And I think the root cause of this issue may on the service side changes.
            Hide
            olblak Olivier Vernin added a comment -

            I confirm that the fix is deployed and working on ci.jenkins.io

            Show
            olblak Olivier Vernin added a comment - I confirm that the fix is deployed and working on ci.jenkins.io
            jieshe Jie Shen made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Released As 0.7.5
            Hide
            psftw Peter Salvatore added a comment -

            Upgraded to 0.7.5 and we are back in business.  Thank you so much Jie Shen !

            Show
            psftw Peter Salvatore added a comment - Upgraded to 0.7.5 and we are back in business.  Thank you so much Jie Shen !
            Hide
            faizan Muhammad Faizan ul haq added a comment -

            Thanks for promot fix but the pain we went through was big and we are kind of ended up in "trust" thing on this plugin.

            Since the API changes can cause plugin to malfunction, could you please set regression tests to run periodically to catch if API changes are causing issues to an existing plugin.

            Any such measures are appreciated. 

            Show
            faizan Muhammad Faizan ul haq added a comment - Thanks for promot fix but the pain we went through was big and we are kind of ended up in "trust" thing on this plugin. Since the API changes can cause plugin to malfunction, could you please set regression tests to run periodically to catch if API changes are causing issues to an existing plugin. Any such measures are appreciated. 
            Hide
            jieshe Jie Shen added a comment -

            Muhammad Faizan ul haq, indeed it is a bug from service side instead of API changes caused this issue. But I will make some effort to try to get alerts from our plugins before users report  issues, thanks. 

            Show
            jieshe Jie Shen added a comment - Muhammad Faizan ul haq , indeed it is a bug from service side instead of API changes caused this issue. But I will make some effort to try to get alerts from our plugins before users report  issues, thanks. 

              People

              • Assignee:
                jieshe Jie Shen
                Reporter:
                faizan Muhammad Faizan ul haq
              • Votes:
                8 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: