Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41568

Failing to provision new agents despite not being at maxumum

    Details

    • Similar Issues:

      Description

      See the attached screenshot. I'm not sure how I ended up getting into this state.

      Here's what I do know:

      • There are not 16 VMs provisioned in this environment despite the System Log claiming there are.
      • There have been recent ARM deployment errors for one of my VM templates:
      {
        "status": "Failed",
        "error": {
          "code": "ResourceDeploymentFailure",
          "message": "The resource operation completed with terminal provisioning state 'Failed'.",
          "details": [
            {
              "code": "VMExtensionProvisioningTimeout",
              "message": "Provisioning of VM extension 'customScriptwin2012-110340' has timed out. Extension installation may be taking too long, or extension status could not be obtained."
            }
          ]
        }
      }
      
      {
        "error": {
          "code": "ResourceNotFound",
          "message": "The Resource 'Microsoft.Compute/virtualMachines/win2012-a13100/extensions/customScriptwin2012-a13100' under resource group 'jenkinsinfra-vmagents' was not found."
        }
      }
      
      {
        "error": {
          "code": "NotFound",
          "target": "vmName",
          "message": "The entity was not found."
        }
      }
      

      My theory is that the template failed to fully initialize, but was counted as a provisioned VM within the plugin's tracking of these things.

        Attachments

          Activity

          Hide
          clguiman Claudiu Guiman added a comment -

          R. Tyler Croy Is this related to JENKINS-41569 ? Are the ghost provisioned images Windows VMs?

          Show
          clguiman Claudiu Guiman added a comment - R. Tyler Croy Is this related to JENKINS-41569 ? Are the ghost provisioned images Windows VMs?
          Hide
          rtyler R. Tyler Croy added a comment -

          I believe they are related, yes

          Show
          rtyler R. Tyler Croy added a comment - I believe they are related, yes
          Hide
          clguiman Claudiu Guiman added a comment -

          The maximum VM count is the number of VMs in the resource groups. Some of these might not be active Jenkins agents.
          I'll fix the code to include only VMs deployed from that specific Jenkins instance, but this will not match with the number of nodes (some agents might be deploying or getting deleted)

          Show
          clguiman Claudiu Guiman added a comment - The maximum VM count is the number of VMs in the resource groups. Some of these might not be active Jenkins agents. I'll fix the code to include only VMs deployed from that specific Jenkins instance, but this will not match with the number of nodes (some agents might be deploying or getting deleted)
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Claudiu Guiman
          Path:
          src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java
          src/main/java/com/microsoft/azure/vmagent/util/AzureUtil.java
          src/main/resources/com/microsoft/azure/vmagent/AzureVMCloud/config.properties
          src/test/java/com/microsoft/azure/vmagent/test/ITAzureVMManagementServiceDelegate.java
          src/test/java/com/microsoft/azure/vmagent/test/IntegrationTest.java
          http://jenkins-ci.org/commit/azure-vm-agents-plugin/0d342f12601405a83587f6d940421b83f5dca721
          Log:
          getVirtualMachineCount counts only the VMs deployed from the current Jenkins instance [FIXED JENKINS-41568]

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Claudiu Guiman Path: src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java src/main/java/com/microsoft/azure/vmagent/util/AzureUtil.java src/main/resources/com/microsoft/azure/vmagent/AzureVMCloud/config.properties src/test/java/com/microsoft/azure/vmagent/test/ITAzureVMManagementServiceDelegate.java src/test/java/com/microsoft/azure/vmagent/test/IntegrationTest.java http://jenkins-ci.org/commit/azure-vm-agents-plugin/0d342f12601405a83587f6d940421b83f5dca721 Log: getVirtualMachineCount counts only the VMs deployed from the current Jenkins instance [FIXED JENKINS-41568]

            People

            • Assignee:
              clguiman Claudiu Guiman
              Reporter:
              rtyler R. Tyler Croy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: