Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41330

Extra VMs left running in Resource Group

    Details

    • Similar Issues:

      Description

      We had some templates fail to deploy on ci.jenkins.io, upon further investigation I discovered that the reason the ARM deploy had failed is because the resource group was exceeding a quota.

      It turns out, Azure has a *lot* more VMs spun up than our Jenkins currently has attached as agents.

      There are *six* VMs running in Azure and basically none are registered as agents in Jenkins. I don't know how long they all have been running

      Not sure what diagnostics could help, i'm going to delete these excess VMs so I can stop burning money.

        Attachments

        1. Selection_050.png
          Selection_050.png
          29 kB
        2. Selection_049.png
          Selection_049.png
          105 kB
        3. Selection_048.png
          Selection_048.png
          203 kB

          Activity

          Hide
          clguiman Claudiu Guiman added a comment -

          I'll update the cleanup task to remove all resources from the resource group, not just the one that got deleted.

          Show
          clguiman Claudiu Guiman added a comment - I'll update the cleanup task to remove all resources from the resource group, not just the one that got deleted.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Claudiu Guiman
          Path:
          src/main/java/com/microsoft/azure/vmagent/AzureVMAgentCleanUpTask.java
          src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java
          src/main/java/com/microsoft/azure/vmagent/util/AzureUtil.java
          src/main/java/com/microsoft/azure/vmagent/util/Constants.java
          src/main/resources/customImageTemplate.json
          src/main/resources/customImageTemplateWithScript.json
          src/main/resources/referenceImageTemplate.json
          src/main/resources/referenceImageTemplateWithScript.json
          src/test/java/com/microsoft/azure/vmagent/test/ITAzureVMAgentCleanUpTask.java
          src/test/java/com/microsoft/azure/vmagent/test/ITAzureVMManagementServiceDelegate.java
          src/test/java/com/microsoft/azure/vmagent/test/IntegrationTest.java
          src/test/java/com/microsoft/azure/vmagent/test/TestDeploymentTag.java
          http://jenkins-ci.org/commit/azure-vm-agents-plugin/2f70f8df6d40f3d400d4b560569982eb51e612a4
          Log:
          Tag deployed resources and remove them if they leak [FIXED JENKINS-41330]

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Claudiu Guiman Path: src/main/java/com/microsoft/azure/vmagent/AzureVMAgentCleanUpTask.java src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java src/main/java/com/microsoft/azure/vmagent/util/AzureUtil.java src/main/java/com/microsoft/azure/vmagent/util/Constants.java src/main/resources/customImageTemplate.json src/main/resources/customImageTemplateWithScript.json src/main/resources/referenceImageTemplate.json src/main/resources/referenceImageTemplateWithScript.json src/test/java/com/microsoft/azure/vmagent/test/ITAzureVMAgentCleanUpTask.java src/test/java/com/microsoft/azure/vmagent/test/ITAzureVMManagementServiceDelegate.java src/test/java/com/microsoft/azure/vmagent/test/IntegrationTest.java src/test/java/com/microsoft/azure/vmagent/test/TestDeploymentTag.java http://jenkins-ci.org/commit/azure-vm-agents-plugin/2f70f8df6d40f3d400d4b560569982eb51e612a4 Log: Tag deployed resources and remove them if they leak [FIXED JENKINS-41330]
          Hide
          clguiman Claudiu Guiman added a comment - - edited

          The fix will make sure there are no leaked deployments provisioned with Azure VM Agents 0.4.2+
          After you upgrade to the latest plugin (there's probably going to be a release next week) please make sure you don't have any older leaked resources (new resource will have this tag key/value: JenkinsManagedTag:ManagedByAzureVMAgents).

          Show
          clguiman Claudiu Guiman added a comment - - edited The fix will make sure there are no leaked deployments provisioned with Azure VM Agents 0.4.2+ After you upgrade to the latest plugin (there's probably going to be a release next week) please make sure you don't have any older leaked resources (new resource will have this tag key/value: JenkinsManagedTag:ManagedByAzureVMAgents).

            People

            • Assignee:
              clguiman Claudiu Guiman
              Reporter:
              rtyler R. Tyler Croy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: