[NCP] Deadline exceeded when terminating VMs in MCI

[yunkon-kim]

cm-beetle                | 9:00AM DBG pkg/client/tumblebug/mci.go:200 > Deleting MCI
cb-tumblebug             | 9:00AM DBG src/core/infra/manageInfo.go:1829 > [Delete MCI] mmci01
cb-tumblebug             | 9:00AM DBG src/core/infra/control.go:60 > [Get MCI requested action: refine
cb-tumblebug             | 9:00AM DBG src/core/infra/control.go:126 > [refine MCI]
cb-tumblebug             | 9:00AM DBG src/core/infra/control.go:147 > [vmInfo.Status] Running
cb-tumblebug             | 9:00AM DBG src/core/infra/control.go:60 > [Get MCI requested action: terminate
cb-tumblebug             | 9:00AM DBG src/core/infra/control.go:92 > [terminate MCI]
cb-tumblebug             | 9:00AM DBG src/core/infra/control.go:358 > [ControlVmAsync] /ns/mig01/mci/mmci01/vm/migrated-00a9f3d4-74b6-e811-906e-000ffee02d5c-1
cb-tumblebug             | 9:09AM ERR src/core/infra/control.go:454 > error="[Error from: http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0] Message: Delete \"http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
cm-beetle                | 9:09AM ERR pkg/client/tumblebug/mci.go:226 > Failed to delete MCI error="[Error from: http://cb-tumblebug:1323/tumblebug/ns/mig01/mci/mmci01?option=terminate] Status code: 500 Internal Server Error, Message: {\"message\":\"[[Error from: http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0] Message: Delete \\\"http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0\\\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]\"}\n"
cm-beetle                | 9:09AM ERR pkg/core/migration/vm-infra.go:398 > failed to delete the infrastructure (nsId: mig01, infraId: mmci01) error="[Error from: http://cb-tumblebug:1323/tumblebug/ns/mig01/mci/mmci01?option=terminate] Status code: 500 Internal Server Error, Message: {\"message\":\"[[Error from: http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0] Message: Delete \\\"http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0\\\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]\"}\n"
cm-beetle                | 9:09AM ERR pkg/api/rest/controller/migration.go:350 > failed to delete the migrated multi-cloud infrastructure error="[Error from: http://cb-tumblebug:1323/tumblebug/ns/mig01/mci/mmci01?option=terminate] Status code: 500 Internal Server Error, Message: {\"message\":\"[[Error from: http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0] Message: Delete \\\"http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0\\\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]\"}\n"
cm-beetle                | 9:09AM INF pkg/api/rest/middlewares/zerologger.go:56 > request URI=/beetle/migration/ns/mig01/mci/mmci01?option=terminate bytes_in= bytes_out=404 id=1754470808423182681 latency_human=10m1.305206566s method=DELETE remote_ip=172.21.0.1 status=500
cb-tumblebug             | Result: {migrated-00a9f3d4-74b6-e811-906e-000ffee02d5c-1  [Error from: http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0] Message: Delete "http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)}
cb-tumblebug             | 9:09AM ERR src/core/infra/manageInfo.go:1856 > error="[[Error from: http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0] Message: Delete \"http://cb-spider:1024/spider/vm/d29hhlfmh8im6se14hm0\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]"
cb-tumblebug             | 9:09AM INF src/interface/rest/server/middlewares/zerologger.go:59 > request ID=1754470808616925560 Method=DELETE URI=/tumblebug/ns/mig01/mci/mmci01?option=terminate clientIP=172.21.0.6 latency=10m1.110615757s status=500

---

(Update - Same issue reoccurred)

• Request to delete VM Infrastructure containing the following VMs
• Deadline exceeded during termination

...
"vm": [
{
"name": "migrated-00a9f3d4-74b6-e811-906e-000ffee02d5c",
"subGroupSize": "",
"label": null,
"description": "a recommended virtual machine 01 for 00a9f3d4-74b6-e811-906e-000ffee02d5c",
"connectionName": "ncpvpc-kr",
"specId": "ncpvpc+kr+m32-g3a",
"imageId": "ncpvpc+23789321",
"vNetId": "mig-vnet-01", 
"subnetId": "mig-subnet-01", 
"securityGroupIds": [ 
"mig-sg-01" 
], 
"sshKeyId": "mig-sshkey-01", 
"dataDiskIds": null 
} 
]
...

[seokho-son]

@yunkon-kim

maybe.. related with the following issue (hypervisor mismatched)

• [NCPVPC:VM] Termination of the VM is stuck in a loop and never completes. cb-spider#1507

[yunkon-kim]

@seokho-son Thank you for sharing the related issue :)

My case may be slightly different from the issue shared. When I retry deleting the VM Infra, it terminates normally and completes the deletion.

I opened the issue like a memo, so the situation wasn't properly displayed ;; 🙏

[yunkon-kim]

Tumblebug has temporarily set the API request timeout to 15 minutes for NCPs.

• Set a longer API request timeout for the control VM in the NCP VPC cb-tumblebug#2093

To address this issue, improvements are planned for Spider and the NCP driver.