Provision failures does not clean up deployments

[timja]

We're seeing a number of failures on ci.jenkins.io which stem from Azure Container Agents failing to provision. It looks like failed ACI deployments don't get cleaned up automatically by the plugin.

ACI is returning the following failure:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.","details":[{"code":"Conflict","message":"{\r\n \"error\": {\r\n \"code\": \"ServiceUnavailable\",\r\n \"message\": \"The requested resource is not available in the location 'eastus2' at this moment. Please retry with a different resource request or in another location. Resource requested: '2' CPU '4' GB memory 'Linux' OS\"\r\n }\r\n}"}]}

Which is annoying, but whatever. The bug here is that the plugin does not clean up this failed deployment, so we have exceeded our quota of Deployments of 800, all failures like this one!

I recall this exact same issue in the Azure VM Agents plugin at one point as well.

[JENKINS-57204] created by rtyler

[timja]

Thanks for reporting this issue. It is similar as GH-48045 in VM agent plugin. I will make a PR soon to fix this.

by jieshe

[timja]

It appears this issue is back on ci.jenkins.io, for some reason, deployments are not cleaned up

2019-10-21 10:50:55.713+0000 [id=55] WARNING hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected exception encountered while provisioning agent aci-maven
java.lang.Exception: Status code 409, {"error":{"code":"DeploymentQuotaExceeded","message":"Creating the deployment 'aci-maven-h3c19t5n' would exceed the quota of '800'. The current deployment count is '800', please delete some deployments before creating a new one. Please see https://aka.ms/arm-debug for usage details."}}
at com.microsoft.jenkins.containeragents.aci.AciService.createDeployment(AciService.java:141)
at com.microsoft.jenkins.containeragents.aci.AciContainerTemplate.provisionAgents(AciContainerTemplate.java:128)
at com.microsoft.jenkins.containeragents.aci.AciCloud$1.call(AciCloud.java:104)
Caused: java.lang.Exception
at com.microsoft.jenkins.containeragents.aci.AciCloud$1.call(AciCloud.java:140)
at com.microsoft.jenkins.containeragents.aci.AciCloud$1.call(AciCloud.java:80)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

by olblak

[timja]

I was trying to fix this at #42 . The fix has been released in 1.1.0. If it does not work, is there any log for why the cleanup action fails?

by jieshe