vNet is not deleted properly

[yunkon-kim]

What happened
: vNet is not deleted properly when calling /migration/ns/{nsId}/mci/{mciId}.
:

What you expected to happen
: vNet and subnets are deleted properly.

Anything else we need to know?
: /migration/ns/{nsId}/resources/vNet/{vNetId} needs to be called separately to delete the vNet and subnets.

Proposed solution
: First, it's necessary to check and debug the target infra deletion mechanism.

[yunkon-kim]

@powerkimhub (cc. @seokho-son)

안녕하세요. vNet/Subnet 리소스가 정상적으로 삭제되지 않는 문제를 분석했으며, 확인을 요청드리고자 합니다.

현황 및 문제점

• Tumblebug의 삭제 절차: Subnet 강제 삭제 요청 후 VPC 강제 삭제를 순차적으로 호출하고 있습니다.
• 발생 문제: 위 절차대로 진행 시, 특정 CSP에서 API 응답이 없거나 예상과 다른 결과(예: 삭제 실패)가 반환되는 현상을 확인했습니다.
• 주요 가설: CSP별로 VPC 및 Subnet의 삭제 로직과 메타데이터 갱신 시점이 달라 발생하는 문제로 추정됩니다. 이로 인해 Tumblebug 상위에서는 삭제 상태를 정확히 인지하기 어렵습니다.

(참고: 현재 이슈로 인해 각 CSP에 삭제되지 않은 VPC 자원이 다수 존재할 수 있습니다;;;)

검토 요청 사항

분석 내용(특히 아래 로그)을 바탕으로, Spider에 다음 항목들에 대한 검토를 부탁드립니다.

1. API 처리 방식의 일관성 (동기/비동기)

• vNet/Subnet 삭제 API가 CSP 드라이버별로 동기/비동기 처리가 혼재되어 있을 수 있습니다. (예: NCP 드라이버의 Subnet 삭제가 동기로 개선됨 by @innodreamer)
• 주요 CSP 드라이버 전반에 걸쳐 삭제 API가 명시적인 완료(성공/실패)를 반환할 때까지 대기하는 동기 방식으로 처리되는지 확인을 요청 드립니다.

2. 메타데이터 정합성

• CSP에서 Subnet 리소스 삭제가 완료된 후, Spider의 메타 정보가 즉시 갱신되지 않는 경우가 있는 것으로 보입니다. (Alibaba 로그 참고)
• 삭제 작업 성공 시, Spider의 메타데이터가 CSP의 실제 상태와 일관성을 유지하는지 확인이 필요합니다.

3. API 응답의 일관성

• 리소스 삭제 API 호출 시, 결과(성공, 실패, 진행 중)를 명확하게 반환하지 않거나 응답 자체가 누락되는 경우가 있습니다. (NCP 로그 참고)
• 모든 드라이버가 삭제 요청에 대해 항상 명확하고 일관된 상태 값을 Tumblebug으로 반환하고 있는지 검토를 부탁드립니다.

Tumblebug 후속 조치 계획

Spider 측면에서 검토 또는 개선이 완료되면, Tumblebug에서는 변경된 Spider의 응답을 보다 안정적으로 처리하기 위한 로직을 보강하도록 하겠습니다.

---

분석 환경 및 삭제 절차

Envrionment

• CM-Beetle v0.3.1+@(트러블슈팅 코드 추가 -> Upstream 반영 후 릴리스 예정)
• CB-Tumblebug v0.11.2+@(Spider API 요청 및 응답에 대한 디버그 로그 추가 -> Upstream 반영 예정)
• CB-Spider v0.11.1

CM-Beetle의 VM Infra 삭제 과정

1. API 요청 수신
2. MCI 조회
3. MCI 삭제
4. Security Groups 삭제
5. SSH Key 삭제
6. vNet 삭제 (action: withsubnets) -> 현 이슈 관련 부분, 아래 CB-Tumblebug vNet 삭제 로직에서 추가 설명
7. Shared resources 삭제
8. API 응답 발송

CB-Tumblebug의 vNet 삭제 과정

1. API 요청 수신
2. vNet 조회 (subnet 정보 포함)
3. Subnet 삭제 (force), (note: subnet 수 만큼 반복)
   1. Subnet 조회
   2. Subnet 정보(object) 업데이트 (status: deleting)
   3. Subnet 삭제 (force) requesting to Spider
   4. Subnet 정보(object) 삭제
   5. vNet 정보(object) 업데이트 (-> 삭제된 Subnets 반영)
4. vNet 재조회
5. vNet 정보(object) 업데이트 (status: deleting)
6. vNet 삭제 (force) requesting to Spider (CSP별로 삭제 로직, 메타데이터 갱신 로직, 등이 상이할 것으로 예상됨)
7. vNet 정보(object) 삭제
8. API 응답 발송

---

테스트 및 분석 결과

🟢 AWS의 VM Infra 삭제 시, 모든 자원 및 Object 삭제 성공

Click to see logs

cm-beetle                | 6:03AM DBG pkg/core/migration/vm-infra.go:460 > Deleting VNets (nsId: mig01, vNets: map[mig-vnet-01:{}])
cm-beetle                | 6:03AM DBG pkg/core/migration/vm-infra.go:464 > Deleting VNet (nsId: mig01, vNetId: mig-vnet-01, action: withsubnets)
cm-beetle                | 6:03AM DBG pkg/client/tumblebug/vnet.go:94 > Deleting Virtual Network
cm-beetle                | 2025/08/06 06:03:02.164048 WARN RESTY Using Basic Auth in HTTP mode is not secure, use HTTPS
cb-tumblebug             | 6:03AM INF src/core/resource/vnet.go:917 > DeleteVNet
cb-tumblebug             | 6:03AM DBG src/core/resource/vnet.go:957 > subnetsKv: [{Key:/ns/mig01/resources/vNet/mig-vnet-01/subnet/mig-subnet-01 Value:{"resourceType":"subnet","id":"mig-subnet-01","uid":"d29emmnja4cgnmskpin0","cspResourceName":"d29emmnja4cgnmskpin0","cspResourceId":"subnet-06e9773e700abc508","name":"mig-subnet-01","connectionName":"aws-ap-northeast-2","connectionConfig":{"configName":"","providerName":"","driverName":"","credentialName":"","credentialHolder":"","regionZoneInfoName":"","regionZoneInfo":{"assignedRegion":"","assignedZone":""},"regionDetail":{"regionId":"","regionName":"","description":"","location":{"display":"","latitude":0,"longitude":0},"zones":null},"regionRepresentative":false,"verified":false},"cspVNetName":"d29emmnja4cgnmskpimg","cspVNetId":"vpc-07df7bbcb0327479a","status":"Available","ipv4_CIDR":"192.168.110.0/24","zone":"ap-northeast-2a","keyValueList":[{"key":"AssignIpv6AddressOnCreation","value":"false"},{"key":"AvailabilityZone","value":"ap-northeast-2a"},{"key":"AvailabilityZoneId","value":"apne2-az1"},{"key":"AvailableIpAddressCount","value":"251"},{"key":"CidrBlock","value":"192.168.110.0/24"},{"key":"DefaultForAz","value":"false"},{"key":"MapCustomerOwnedIpOnLaunch","value":"false"},{"key":"MapPublicIpOnLaunch","value":"false"},{"key":"OwnerId","value":"635484366616"},{"key":"State","value":"available"},{"key":"SubnetArn","value":"arn:aws:ec2:ap-northeast-2:635484366616:subnet/subnet-06e9773e700abc508"},{"key":"SubnetId","value":"subnet-06e9773e700abc508"},{"key":"Tags","value":"{Key:Name,Value:d29emmnja4cgnmskpin0}"},{"key":"VpcId","value":"vpc-07df7bbcb0327479a"}],"description":""}}]
cb-tumblebug             | 6:03AM INF src/core/resource/subnet.go:620 > DeleteSubnet
cb-tumblebug             | 6:03AM DBG src/core/resource/subnet.go:740 > [Request to Spider] Deleting Subnet (url: http://cb-spider:1024/spider/vpc/d29emmnja4cgnmskpimg/subnet/d29emmnja4cgnmskpin0, request body: {ConnectionName:aws-ap-northeast-2})
cb-tumblebug             | 6:03AM DBG src/core/resource/subnet.go:758 > [Response from Spider] Deleting Subnet (response body: {Result:true})
cb-tumblebug             | 6:03AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/subnet/d29emmnja4cgnmskpin0
cb-tumblebug             | 6:03AM DBG src/core/resource/subnet.go:816 > subnet (mig-subnet-01) has been deleted
cb-tumblebug             | 6:03AM DBG src/core/resource/vnet.go:1044 > [Request to Spider] Deleting VPC (url: http://cb-spider:1024/spider/vpc/d29emmnja4cgnmskpimg, request body: {ConnectionName:aws-ap-northeast-2})
cb-tumblebug             | 6:03AM DBG src/core/resource/vnet.go:1062 > [Response from Spider] Deleting VPC (response body: {Result:true})
cb-tumblebug             | 6:03AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/vNet/d29emmnja4cgnmskpimg
cb-tumblebug             | 6:03AM DBG src/core/resource/vnet.go:1097 > vNet (mig-vnet-01) has been deleted
cm-beetle                | 6:03AM DBG pkg/client/tumblebug/vnet.go:124 > Deleted VNet (vNetId: mig-vnet-01) successfully
cm-beetle                | 6:03AM DBG pkg/core/migration/vm-infra.go:470 > VNet deleted (nsId: mig01, vNetId: mig-vnet-01, msg:{the vNet (mig-vnet-01) has been deleted})

---

🟢Azure의 VM Infra 삭제 시, 모든 자원 및 Object 삭제 성공

Click to see logs

cm-beetle                | 6:22AM DBG pkg/core/migration/vm-infra.go:460 > Deleting VNets (nsId: mig01, vNets: map[mig-vnet-01:{}])
cm-beetle                | 6:22AM DBG pkg/core/migration/vm-infra.go:464 > Deleting VNet (nsId: mig01, vNetId: mig-vnet-01, action: withsubnets)
cm-beetle                | 6:22AM DBG pkg/client/tumblebug/vnet.go:94 > Deleting Virtual Network
cb-tumblebug             | 6:22AM INF src/core/resource/vnet.go:917 > DeleteVNet
cb-tumblebug             | 6:22AM DBG src/core/resource/vnet.go:957 > subnetsKv: [{Key:/ns/mig01/resources/vNet/mig-vnet-01/subnet/mig-subnet-01 Value:{"resourceType":"subnet","id":"mig-subnet-01","uid":"d29f2n7ja4cgnmskpir0","cspResourceName":"d29f2n7ja4cgnmskpir0","cspResourceId":"/subscriptions/a20fed83-96bd-4480-92a9-140b8e3b7c3a/resourceGroups/koreacentral/providers/Microsoft.Network/virtualNetworks/d29f2n7ja4cgnmskpiqg/subnets/d29f2n7ja4cgnmskpir0","name":"mig-subnet-01","connectionName":"azure-koreacentral","connectionConfig":{"configName":"","providerName":"","driverName":"","credentialName":"","credentialHolder":"","regionZoneInfoName":"","regionZoneInfo":{"assignedRegion":"","assignedZone":""},"regionDetail":{"regionId":"","regionName":"","description":"","location":{"display":"","latitude":0,"longitude":0},"zones":null},"regionRepresentative":false,"verified":false},"cspVNetName":"d29f2n7ja4cgnmskpiqg","cspVNetId":"/subscriptions/a20fed83-96bd-4480-92a9-140b8e3b7c3a/resourceGroups/koreacentral/providers/Microsoft.Network/virtualNetworks/d29f2n7ja4cgnmskpiqg","status":"Available","ipv4_CIDR":"192.168.110.0/24","keyValueList":[{"key":"ID","value":"/subscriptions/a20fed83-96bd-4480-92a9-140b8e3b7c3a/resourceGroups/koreacentral/providers/Microsoft.Network/virtualNetworks/d29f2n7ja4cgnmskpiqg/subnets/d29f2n7ja4cgnmskpir0"},{"key":"Name","value":"d29f2n7ja4cgnmskpir0"},{"key":"Properties","value":"{addressPrefix:192.168.110.0/24,delegations:[],privateEndpointNetworkPolicies:Disabled,privateLinkServiceNetworkPolicies:Enabled,provisioningState:Succeeded,serviceEndpoints:[{locations:[koreacentral,koreasouth],provisioningState:Succeeded,service:Microsoft.Storage}]}"},{"key":"Type","value":"Microsoft.Network/virtualNetworks/subnets"},{"key":"Etag","value":"W/\\693a4362-f9f8-4ada-8f7d-0c02f2e0f7ef\\"}],"description":""}}]
cb-tumblebug             | 6:22AM INF src/core/resource/subnet.go:620 > DeleteSubnet
cb-tumblebug             | 6:22AM DBG src/core/resource/subnet.go:740 > [Request to Spider] Deleting Subnet (url: http://cb-spider:1024/spider/vpc/d29f2n7ja4cgnmskpiqg/subnet/d29f2n7ja4cgnmskpir0, request body: {ConnectionName:azure-koreacentral})
cb-tumblebug             | 6:23AM DBG src/core/resource/subnet.go:758 > [Response from Spider] Deleting Subnet (response body: {Result:true})
cb-tumblebug             | 6:23AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/subnet/d29f2n7ja4cgnmskpir0
cb-tumblebug             | 6:23AM DBG src/core/resource/subnet.go:816 > subnet (mig-subnet-01) has been deleted
cb-tumblebug             | 6:23AM DBG src/core/resource/vnet.go:1044 > [Request to Spider] Deleting VPC (url: http://cb-spider:1024/spider/vpc/d29f2n7ja4cgnmskpiqg, request body: {ConnectionName:azure-koreacentral})
cb-tumblebug             | 6:23AM DBG src/core/resource/vnet.go:1062 > [Response from Spider] Deleting VPC (response body: {Result:true})
cb-tumblebug             | 6:23AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/vNet/d29f2n7ja4cgnmskpiqg
cb-tumblebug             | 6:23AM DBG src/core/resource/vnet.go:1097 > vNet (mig-vnet-01) has been deleted
cm-beetle                | 6:23AM DBG pkg/client/tumblebug/vnet.go:124 > Deleted VNet (vNetId: mig-vnet-01) successfully
cm-beetle                | 6:23AM DBG pkg/core/migration/vm-infra.go:470 > VNet deleted (nsId: mig01, vNetId: mig-vnet-01, msg:{the vNet (mig-vnet-01) has been deleted})

---

🟢GCP의 VM Infra 삭제 시, 모든 자원 및 Object 삭제 성공

Click to see logs

cm-beetle                | 6:43AM DBG pkg/core/migration/vm-infra.go:460 > Deleting VNets (nsId: mig01, vNets: map[mig-vnet-01:{}])
cm-beetle                | 6:43AM DBG pkg/core/migration/vm-infra.go:464 > Deleting VNet (nsId: mig01, vNetId: mig-vnet-01, action: withsubnets)
cm-beetle                | 6:43AM DBG pkg/client/tumblebug/vnet.go:94 > Deleting Virtual Network
cb-tumblebug             | 6:43AM INF src/core/resource/vnet.go:917 > DeleteVNet
cb-tumblebug             | 6:43AM DBG src/core/resource/vnet.go:957 > subnetsKv: [{Key:/ns/mig01/resources/vNet/mig-vnet-01/subnet/mig-subnet-01 Value:{"resourceType":"subnet","id":"mig-subnet-01","uid":"d29fakvja4cgnmskpiv0","cspResourceName":"d29fakvja4cgnmskpiv0","cspResourceId":"d29fakvja4cgnmskpiv0","name":"mig-subnet-01","connectionName":"gcp-asia-northeast3","connectionConfig":{"configName":"","providerName":"","driverName":"","credentialName":"","credentialHolder":"","regionZoneInfoName":"","regionZoneInfo":{"assignedRegion":"","assignedZone":""},"regionDetail":{"regionId":"","regionName":"","description":"","location":{"display":"","latitude":0,"longitude":0},"zones":null},"regionRepresentative":false,"verified":false},"cspVNetName":"d29fakvja4cgnmskpiug","cspVNetId":"d29fakvja4cgnmskpiug","status":"Available","ipv4_CIDR":"192.168.110.0/24","keyValueList":[{"key":"CreationTimestamp","value":"2025-08-05T23:25:39.440-07:00"},{"key":"EnableFlowLogs","value":"false"},{"key":"Fingerprint","value":"zChfKZpMMkc="},{"key":"GatewayAddress","value":"192.168.110.1"},{"key":"Id","value":"7440856669568846732"},{"key":"IpCidrRange","value":"192.168.110.0/24"},{"key":"Kind","value":"compute#subnetwork"},{"key":"Name","value":"d29fakvja4cgnmskpiv0"},{"key":"Network","value":"https://www.googleapis.com/compute/v1/projects/ykkim-etri/global/networks/d29fakvja4cgnmskpiug"},{"key":"PrivateIpGoogleAccess","value":"false"},{"key":"PrivateIpv6GoogleAccess","value":"DISABLE_GOOGLE_ACCESS"},{"key":"Purpose","value":"PRIVATE"},{"key":"Region","value":"https://www.googleapis.com/compute/v1/projects/ykkim-etri/regions/asia-northeast3"},{"key":"SelfLink","value":"https://www.googleapis.com/compute/v1/projects/ykkim-etri/regions/asia-northeast3/subnetworks/d29fakvja4cgnmskpiv0"},{"key":"StackType","value":"IPV4_ONLY"}],"description":""}}]
cb-tumblebug             | 6:43AM INF src/core/resource/subnet.go:620 > DeleteSubnet
cb-tumblebug             | 6:43AM DBG src/core/resource/subnet.go:740 > [Request to Spider] Deleting Subnet (url: http://cb-spider:1024/spider/vpc/d29fakvja4cgnmskpiug/subnet/d29fakvja4cgnmskpiv0, request body: {ConnectionName:gcp-asia-northeast3})
cb-tumblebug             | 6:43AM DBG src/core/resource/subnet.go:758 > [Response from Spider] Deleting Subnet (response body: {Result:true})
cb-tumblebug             | 6:43AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/subnet/d29fakvja4cgnmskpiv0
cb-tumblebug             | 6:43AM DBG src/core/resource/subnet.go:816 > subnet (mig-subnet-01) has been deleted
cb-tumblebug             | 6:43AM DBG src/core/resource/vnet.go:1044 > [Request to Spider] Deleting VPC (url: http://cb-spider:1024/spider/vpc/d29fakvja4cgnmskpiug, request body: {ConnectionName:gcp-asia-northeast3})
cb-tumblebug             | 6:43AM DBG src/core/resource/vnet.go:1062 > [Response from Spider] Deleting VPC (response body: {Result:true})
cb-tumblebug             | 6:43AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/vNet/d29fakvja4cgnmskpiug
cb-tumblebug             | 6:43AM DBG src/core/resource/vnet.go:1097 > vNet (mig-vnet-01) has been deleted
cm-beetle                | 6:43AM DBG pkg/client/tumblebug/vnet.go:124 > Deleted VNet (vNetId: mig-vnet-01) successfully
cm-beetle                | 6:43AM DBG pkg/core/migration/vm-infra.go:470 > VNet deleted (nsId: mig01, vNetId: mig-vnet-01, msg:{the vNet (mig-vnet-01) has been deleted})

---

🔴 Alibaba의 VM Infra 삭제 시, vNet/Subnet 삭제 실패

분석 결과
: CB-Tumblebug에서 Subnet을 삭제(via Spider) 완료한 이후, vNet (i.e., VPC)를 삭제 요청(to Spider)을 했을 때
: Spider 내부에서 이미 삭제된 Subnet들을 다시 삭제 시도하려는 것 같음(IncorrectVSwitchId)
: false를 응답, {Result:false}
: (참고) cb-tumblebug | 5:10AM DBG src/core/resource/vnet.go:1054 > [Response from Spider] Deleting VPC (response body: {Result:false})

Click to see logs

cm-beetle                | 5:10AM DBG pkg/core/migration/vm-infra.go:460 > Deleting VNets (nsId: mig01, vNets: map[mig-vnet-01:{}])
cm-beetle                | 5:10AM DBG pkg/core/migration/vm-infra.go:464 > Deleting VNet (nsId: mig01, vNetId: mig-vnet-01, action: withsubnets)
cm-beetle                | 5:10AM DBG pkg/client/tumblebug/vnet.go:94 > Deleting Virtual Network
cb-tumblebug             | 5:10AM INF src/core/resource/vnet.go:909 > DeleteVNet
cb-tumblebug             | 5:10AM DBG src/core/resource/vnet.go:949 > subnetsKv: [{Key:/ns/mig01/resources/vNet/mig-vnet-01/subnet/mig-subnet-01 Value:{"resourceType":"subnet","id":"mig-subnet-01","uid":"d29e6mmhouukv0m4dvig","cspResourceName":"d29e6mmhouukv0m4dvig","cspResourceId":"vsw-mj7bk43ra22or5k3fz27b","name":"mig-subnet-01","connectionName":"alibaba-ap-northeast-2","connectionConfig":{"configName":"","providerName":"","driverName":"","credentialName":"","credentialHolder":"","regionZoneInfoName":"","regionZoneInfo":{"assignedRegion":"","assignedZone":""},"regionDetail":{"regionId":"","regionName":"","description":"","location":{"display":"","latitude":0,"longitude":0},"zones":null},"regionRepresentative":false,"verified":false},"cspVNetName":"d29e6mmhouukv0m4dvi0","cspVNetId":"vpc-mj7ipkigd7l6qnlucjf8m","status":"Available","ipv4_CIDR":"192.168.110.0/24","zone":"ap-northeast-2a","keyValueList":[{"key":"VpcId","value":"vpc-mj7ipkigd7l6qnlucjf8m"},{"key":"Status","value":"Pending"},{"key":"CreationTime","value":"2025-08-06T05:08:45Z"},{"key":"IsDefault","value":"false"},{"key":"AvailableIpAddressCount","value":"252"},{"key":"OwnerId","value":"5881385118576570"},{"key":"VSwitchId","value":"vsw-mj7bk43ra22or5k3fz27b"},{"key":"CidrBlock","value":"192.168.110.0/24"},{"key":"ResourceGroupId","value":"rg-acfnyca3hcx27oq"},{"key":"ZoneId","value":"ap-northeast-2a"},{"key":"VSwitchName","value":"d29e6mmhouukv0m4dvig"},{"key":"EnabledIpv6","value":"false"},{"key":"RouteTable","value":"{ResourceGroupId:,CreationTime:,Status:,RouteTableType:System,VRouterId:,RouteTableId:vtb-mj70o702rgwg2cvswymso,VSwitchIds:{VSwitchId:null},RouteEntrys:{RouteEntry:null}}"},{"key":"Tags","value":"{Tag:null}"}],"description":""}}]
cb-tumblebug             | 5:10AM INF src/core/resource/subnet.go:611 > DeleteSubnet
cb-tumblebug             | 5:10AM DBG src/core/resource/subnet.go:731 > [Request to Spider] Deleting Subnet (url: http://cb-spider:1024/spider/vpc/d29e6mmhouukv0m4dvi0/subnet/d29e6mmhouukv0m4dvig, request body: {ConnectionName:alibaba-ap-northeast-2})
cb-tumblebug             | 5:10AM DBG src/core/resource/subnet.go:749 > [Response from Spider] Deleting Subnet (response body: {Result:true})
cb-tumblebug             | 5:10AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/subnet/d29e6mmhouukv0m4dvig
cb-tumblebug             | 5:10AM DBG src/core/resource/subnet.go:807 > subnet (mig-subnet-01) has been deleted
cb-tumblebug             | 5:10AM DBG src/core/resource/vnet.go:1038 > [Request to Spider] Deleting VPC (url: http://cb-spider:1024/spider/vpc/d29e6mmhouukv0m4dvi0, request body: {ConnectionName:alibaba-ap-northeast-2})
cb-spider                | [CB-SPIDER].[ERROR]: 2025-08-06 05:10:31 VPCHandler.go:490, github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/alibaba/resources.(*AlibabaVPCHandler).DeleteSubnet() - SDK.ServerError
cb-spider                | ErrorCode: IncorrectVSwitchId
cb-spider                | Recommend: https://api.aliyun.com/troubleshoot?q=IncorrectVSwitchId&product=Vpc&requestId=DF9A8315-347A-3BE5-B12F-9FE3FF00FF25
cb-spider                | RequestId: DF9A8315-347A-3BE5-B12F-9FE3FF00FF25
cb-spider                | Message: Specified vSwtich instance is not found during access authentication.
cb-spider                | RespHeaders: map[Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[*] Connection:[keep-alive] Content-Length:[335] Content-Type:[application/json;charset=utf-8] Date:[Wed, 06 Aug 2025 05:10:30 GMT] Keep-Alive:[timeout=25] X-Acs-Request-Id:[DF9A8315-347A-3BE5-B12F-9FE3FF00FF25] X-Acs-Trace-Id:[01dc7222d1db83c1c147c934a2ab7716]] 
cb-spider                | [CB-SPIDER].[ERROR]: 2025-08-06 05:10:31 VPC-SubnetManager.go:1517, github.com/cloud-barista/cb-spider/api-runtime/common-runtime.DeleteVPC() - SDK.ServerError
cb-spider                | ErrorCode: IncorrectVSwitchId
cb-spider                | Recommend: https://api.aliyun.com/troubleshoot?q=IncorrectVSwitchId&product=Vpc&requestId=DF9A8315-347A-3BE5-B12F-9FE3FF00FF25
cb-spider                | RequestId: DF9A8315-347A-3BE5-B12F-9FE3FF00FF25
cb-spider                | Message: Specified vSwtich instance is not found during access authentication.
cb-spider                | RespHeaders: map[Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[*] Connection:[keep-alive] Content-Length:[335] Content-Type:[application/json;charset=utf-8] Date:[Wed, 06 Aug 2025 05:10:30 GMT] Keep-Alive:[timeout=25] X-Acs-Request-Id:[DF9A8315-347A-3BE5-B12F-9FE3FF00FF25] X-Acs-Trace-Id:[01dc7222d1db83c1c147c934a2ab7716]] 
cb-tumblebug             | 5:10AM DBG src/core/resource/vnet.go:1054 > [Response from Spider] Deleting VPC (response body: {Result:false})
cb-tumblebug             | 5:10AM ERR src/core/resource/vnet.go:1067 > error="failed to delete the vNet (mig-vnet-01)"
cb-tumblebug             | 5:10AM ERR src/interface/rest/server/resource/vnet.go:215 > error="failed to delete the vNet (mig-vnet-01)"
cm-beetle                | 5:10AM ERR pkg/client/tumblebug/vnet.go:120 > Failed to delete VNet error="[Error from: http://cb-tumblebug:1323/tumblebug/ns/mig01/resources/vNet/mig-vnet-01?action=withsubnets] Status code: 500 Internal Server Error, Message: {\"message\":\"failed to delete the vNet (mig-vnet-01)\"}\n"
cm-beetle                | 5:10AM ERR pkg/core/migration/vm-infra.go:467 > failed to delete VNet (nsId: mig01, vNetId: mig-vnet-01) error="[Error from: http://cb-tumblebug:1323/tumblebug/ns/mig01/resources/vNet/mig-vnet-01?action=withsubnets] Status code: 500 Internal Server Error, Message: {\"message\":\"failed to delete the vNet (mig-vnet-01)\"}\n"

---

🔴 NCP(VPC)의 VM Infra 삭제 시, vNet/Subnet 삭제 실패

분석 결과
: CB-Tumblebug에서 Subnet을 삭제(via Spider) 완료한 이후, vNet (i.e., VPC)를 삭제 요청(to Spider)을 했을 때
: Spider 내부에서 삭제 할 수 없는 상태로 인식하여 삭제하지 못함 ("The status is non-deletionable.")
: 응답 값 없음 {Result:}
: (참고) cb-tumblebug | 5:30AM DBG src/core/resource/vnet.go:1062 > [Response from Spider] Deleting VPC (response body: {Result:})

Click to see logs

cm-beetle                | 5:30AM DBG pkg/core/migration/vm-infra.go:460 > Deleting VNets (nsId: mig01, vNets: map[mig-vnet-01:{}])
cm-beetle                | 5:30AM DBG pkg/core/migration/vm-infra.go:464 > Deleting VNet (nsId: mig01, vNetId: mig-vnet-01, action: withsubnets)
cm-beetle                | 5:30AM DBG pkg/client/tumblebug/vnet.go:94 > Deleting Virtual Network
cm-beetle                | 2025/08/06 05:30:46.008596 WARN RESTY Using Basic Auth in HTTP mode is not secure, use HTTPS
cb-tumblebug             | 5:30AM INF src/core/resource/vnet.go:917 > DeleteVNet
cb-tumblebug             | 5:30AM DBG src/core/resource/vnet.go:957 > subnetsKv: [{Key:/ns/mig01/resources/vNet/mig-vnet-01/subnet/mig-subnet-01 Value:{"resourceType":"subnet","id":"mig-subnet-01","uid":"d29eao6houukv0m4dvmg","cspResourceName":"d29eao6houukv0m4dvmg","cspResourceId":"247690","name":"mig-subnet-01","connectionName":"ncpvpc-kr","connectionConfig":{"configName":"","providerName":"","driverName":"","credentialName":"","credentialHolder":"","regionZoneInfoName":"","regionZoneInfo":{"assignedRegion":"","assignedZone":""},"regionDetail":{"regionId":"","regionName":"","description":"","location":{"display":"","latitude":0,"longitude":0},"zones":null},"regionRepresentative":false,"verified":false},"cspVNetName":"d29eao6houukv0m4dvm0","cspVNetId":"117318","status":"Available","ipv4_CIDR":"192.168.110.0/24","zone":"KR-1","keyValueList":[{"key":"SubnetNo","value":"247690"},{"key":"VpcNo","value":"117318"},{"key":"ZoneCode","value":"KR-1"},{"key":"SubnetName","value":"d29eao6houukv0m4dvmg"},{"key":"Subnet","value":"192.168.110.0/24"},{"key":"SubnetStatus","value":"{code:RUN,codeName:운영중}"},{"key":"CreateDate","value":"2025-08-06T14:17:31+0900"},{"key":"SubnetType","value":"{code:PUBLIC,codeName:Public}"},{"key":"UsageType","value":"{code:GEN,codeName:General}"},{"key":"NetworkAclNo","value":"161180"}],"description":""}}]
cb-tumblebug             | 5:30AM INF src/core/resource/subnet.go:620 > DeleteSubnet
cb-tumblebug             | 5:30AM DBG src/core/resource/subnet.go:740 > [Request to Spider] Deleting Subnet (url: http://cb-spider:1024/spider/vpc/d29eao6houukv0m4dvm0/subnet/d29eao6houukv0m4dvmg, request body: {ConnectionName:ncpvpc-kr})
cb-tumblebug             | 5:30AM DBG src/core/resource/subnet.go:758 > [Response from Spider] Deleting Subnet (response body: {Result:true})
cb-tumblebug             | 5:30AM INF src/core/common/label/label.go:126 > Label object successfully deleted from kvstore labelKey=/label/subnet/d29eao6houukv0m4dvmg
cb-tumblebug             | 5:30AM DBG src/core/resource/subnet.go:816 > subnet (mig-subnet-01) has been deleted
cb-tumblebug             | 5:30AM DBG src/core/resource/vnet.go:1044 > [Request to Spider] Deleting VPC (url: http://cb-spider:1024/spider/vpc/d29eao6houukv0m4dvm0, request body: {ConnectionName:ncpvpc-kr})
cb-spider                | [CB-SPIDER].[ERROR]: 2025-08-06 05:30:46 VPCHandler.go:300, github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/ncpvpc/resources.(*NcpVpcVPCHandler).DeleteVPC() - Failed to Remove the Subnet with the SystemId : [247690] : [Status: 400 Bad Request, Body: {
cb-spider                |   "responseError": {
cb-spider                |     "returnCode": "1001008",
cb-spider                |     "returnMessage": "The status is non-deletionable."
cb-spider                |   }
cb-spider                | }] 
cb-spider                | [CB-SPIDER].[ERROR]: 2025-08-06 05:30:46 CommonNcpVpcFunc.go:71, github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/ncpvpc/resources.LoggingError() - Failed to Remove the Subnet with the SystemId : [247690] : [Status: 400 Bad Request, Body: {
cb-spider                |   "responseError": {
cb-spider                |     "returnCode": "1001008",
cb-spider                |     "returnMessage": "The status is non-deletionable."
cb-spider                |   }
cb-spider                | }] 
cb-spider                | [CB-SPIDER].[ERROR]: 2025-08-06 05:30:46 VPC-SubnetManager.go:1517, github.com/cloud-barista/cb-spider/api-runtime/common-runtime.DeleteVPC() - Failed to Remove the Subnet with the SystemId : [247690] : [Status: 400 Bad Request, Body: {
cb-spider                |   "responseError": {
cb-spider                |     "returnCode": "1001008",
cb-spider                |     "returnMessage": "The status is non-deletionable."
cb-spider                |   }
cb-spider                | }] 
cb-tumblebug             | 5:30AM DBG src/core/resource/vnet.go:1062 > [Response from Spider] Deleting VPC (response body: {Result:})
cb-tumblebug             | 5:30AM ERR src/core/resource/vnet.go:1065 > error="[Error from: http://cb-spider:1024/spider/vpc/d29eao6houukv0m4dvm0] Status code: 500 Internal Server Error, Message: {\"message\":\"Failed to Remove the Subnet with the SystemId : [247690] : [Status: 400 Bad Request, Body: {\\n  \\\"responseError\\\": {\\n    \\\"returnCode\\\": \\\"1001008\\\",\\n    \\\"returnMessage\\\": \\\"The status is non-deletionable.\\\"\\n  }\\n}]\"}\n"
cb-tumblebug             | 5:30AM ERR src/interface/rest/server/resource/vnet.go:215 > error="[Error from: http://cb-spider:1024/spider/vpc/d29eao6houukv0m4dvm0] Status code: 500 Internal Server Error, Message: {\"message\":\"Failed to Remove the Subnet with the SystemId : [247690] : [Status: 400 Bad Request, Body: {\\n  \\\"responseError\\\": {\\n    \\\"returnCode\\\": \\\"1001008\\\",\\n    \\\"returnMessage\\\": \\\"The status is non-deletionable.\\\"\\n  }\\n}]\"}\n"
cm-beetle                | 5:30AM ERR pkg/client/tumblebug/vnet.go:120 > Failed to delete VNet error="[Error from: http://cb-tumblebug:1323/tumblebug/ns/mig01/resources/vNet/mig-vnet-01?action=withsubnets] Status code: 500 Internal Server Error, Message: {\"message\":\"[Error from: http://cb-spider:1024/spider/vpc/d29eao6houukv0m4dvm0] Status code: 500 Internal Server Error, Message: {\\\"message\\\":\\\"Failed to Remove the Subnet with the SystemId : [247690] : [Status: 400 Bad Request, Body: {\\\\n  \\\\\\\"responseError\\\\\\\": {\\\\n    \\\\\\\"returnCode\\\\\\\": \\\\\\\"1001008\\\\\\\",\\\\n    \\\\\\\"returnMessage\\\\\\\": \\\\\\\"The status is non-deletionable.\\\\\\\"\\\\n  }\\\\n}]\\\"}\\n\"}\n"
cm-beetle                | 5:30AM ERR pkg/core/migration/vm-infra.go:467 > failed to delete VNet (nsId: mig01, vNetId: mig-vnet-01) error="[Error from: http://cb-tumblebug:1323/tumblebug/ns/mig01/resources/vNet/mig-vnet-01?action=withsubnets] Status code: 500 Internal Server Error, Message: {\"message\":\"[Error from: http://cb-spider:1024/spider/vpc/d29eao6houukv0m4dvm0] Status code: 500 Internal Server Error, Message: {\\\"message\\\":\\\"Failed to Remove the Subnet with the SystemId : [247690] : [Status: 400 Bad Request, Body: {\\\\n  \\\\\\\"responseError\\\\\\\": {\\\\n    \\\\\\\"returnCode\\\\\\\": \\\\\\\"1001008\\\\\\\",\\\\n    \\\\\\\"returnMessage\\\\\\\": \\\\\\\"The status is non-deletionable.\\\\\\\"\\\\n  }\\\\n}]\\\"}\\n\"}\n"

[innodreamer]

@yunkon-kim (@powerkimhub)
위의 NCP VPC log를 확인해보니, VPC에 속해있는 각각의 subnet를 삭제한 뒤 VPC를 삭제하는 순서로 vNet 삭제 과정이 진행됩니다.

• Spider 공통 인터페이스 기준으로, RemoveSubnet()으로 소속된 subnet 삭제 후, DeleteVPC()로 VPC가 삭제됨.

NCP VPC 드라이버 기준으로, NCP 경우 subnet 삭제가 비동기로 처리되어 DeleteVPC() 내부에는 subnet 삭제가 완료될때까지 status를 체크하는 waiting time이 이미 추가되어있지만, RemoveSubnet()에는 waiting time이 적용되어 있지 않아 추가했습니다.

• Related PR : [NCP VPC] Add Waiting time to Removesubnet() to Check the Status until the Subnet is Completely Deleted cb-spider#1525

[yunkon-kim]

안정성을 올리는 차원에서, 일단 CB-Tumblebug의 vNet/Subnet operation 로직을 개선해둔 상태 입니다.

• ref) Improve VNet/Subnet operations with enhanced logging and retry logic cb-tumblebug#2078

[yunkon-kim]

@powerkimhub (cc. @seokho-son)

죄송합니다. 착각했던 부분이 있어서 정정드립니다. vNet/subnet 삭제 시, force 옵션을 지정하고 있지 않고 있었습니다. 위 본문에 반영해 놓았습니다.

[yunkon-kim]

Tumblebug에서 서브넷 삭제에 대한 응답을 수신 한 후, 삭제가 완료되었는지 확인하는 로직이 추가되었습니다.

• Ref. feat: add subnet deletion verification and enable debug logging cb-tumblebug#2081

[powerkimhub]

• GCP 관련해서는 다음을 참고해주세요.
• GCP VPC is not deleted frequently cb-tumblebug#2135 (comment)

[powerkimhub]

1. API 처리 방식의 일관성 (동기/비동기)

• VPC/Subnet, SG, KeyPair, VM: Sync Call

• Disk, MyImage, NLB, K8S: ASync Call

  • 즉, API 호출 성공은 API 요청이 서버에 잘 전달됨을 의미하며, 자원의 상태는 보장하지 않음.
  • 사용자는 생성 이후 사용하기 전에 자원의 상태가 Available할 때까지 상태 체크 필요
  • 참고 Disk Status:
    • https://github.com/cloud-barista/cb-spider/blob/2bc8d557deb45b18dcaf2820da69c9fa956370a7/cloud-control-manager/cloud-driver/interfaces/resources/DiskHandler.go#L19-L23
  • 참고 MyImage Status:
    • https://github.com/cloud-barista/cb-spider/blob/2bc8d557deb45b18dcaf2820da69c9fa956370a7/cloud-control-manager/cloud-driver/interfaces/resources/MyImageHandler.go#L19-L20
  • 참고 NLB Status: health 정보 확인, Port 상태 확인
    • https://github.com/cloud-barista/cb-spider/blob/2bc8d557deb45b18dcaf2820da69c9fa956370a7/cloud-control-manager/cloud-driver/interfaces/resources/NLBHandler.go#L91
    • check tcp port: https://cloud-barista.github.io/api/?url=https://raw.githubusercontent.com/cloud-barista/cb-spider/master/api/swagger.yaml#/%5BUtility%5D/check-tcp-port
    • check udp port : https://cloud-barista.github.io/api/?url=https://raw.githubusercontent.com/cloud-barista/cb-spider/master/api/swagger.yaml#/%5BUtility%5D/check-udp-port
  • 참고 K8S Status
    • https://github.com/cloud-barista/cb-spider/blob/2bc8d557deb45b18dcaf2820da69c9fa956370a7/cloud-control-manager/cloud-driver/interfaces/resources/ClusterHandler.go#L19-L33

• ※ dependency 관계가 있는 자원 삭제 관련 특이사항

  • Spideer의 Sync Call 제공에는 한계가 있음.
  • 예를 들면,
  • vpc1에 속하는 sg1을 CSP API 수준에서 완전히 삭제한 후에
  • vpc1을 삭제 시에 한동안 사용 중이므로 삭제 불가능하다는 메시지가 뜨는 경우가 존재합니다.
    • 잠시 후에 다시 시도하면 삭제 가능합니다. (백앤드 자원 가비지 컬렉터 같은...)
    • 이 경우 Spider 입장에서는 사용자의 또 다른 sg2가 실제로(정상적인 상황) vpc1을 사용 중일 수 있으므로
    • vpc1에 대한 삭제를 반복 시도하지 않고 바로 에러를 반환합니다.
    • 이때, 가급적 에러 원인인 의존 관계 자원에 대한 정보가 CSP로부터 올라오면 에러 메시지에 메달아 올려주고 있습니다.
      • (CSP마다, 자원마다, Driver마다 다를 수 있음)
    • 결국, 삭제시 dependency 오류 또는 다른 오류 발생 시에 몇번의 재시도가 필요할 수 있습니다. (안전 장치)
      • TB API를 사용한다면, 이미 포함되어 있을 수 있음(@seokho-son )

• 조만간 Alibaba 등 VPC/Subnet 중심으로 추가 시험 예정입니다.

[seokho-son]

Spideer의 Sync Call 제공에는 한계가 있음.
예를 들면,
vpc1에 속하는 sg1을 CSP API 수준에서 완전히 삭제한 후에
vpc1을 삭제 시에 한동안 사용 중이므로 삭제 불가능하다는 메시지가 뜨는 경우가 존재합니다.
잠시 후에 다시 시도하면 삭제 가능합니다. (백앤드 자원 가비지 컬렉터 같은...)
이 경우 Spider 입장에서는 사용자의 또 다른 sg2가 실제로(정상적인 상황) vpc1을 사용 중일 수 있으므로
vpc1에 대한 삭제를 반복 시도하지 않고 바로 에러를 반환합니다.
이때, 가급적 에러 원인인 의존 관계 자원에 대한 정보가 CSP로부터 올라오면 에러 메시지에 메달아 올려주고 있습니다.
(CSP마다, 자원마다, Driver마다 다를 수 있음)
결국, 삭제시 dependency 오류 또는 다른 오류 발생 시에 몇번의 재시도가 필요할 수 있습니다. (안전 장치)
TB API를 사용한다면, 이미 포함되어 있을 수 있음(@seokho-son )

@powerkimhub
아시다시피, 관련해서 TB는 가급적 Spider 및 드라이버에서 복잡한 오퍼레이션 (재시도 반복 등)을 하지 않는 것을 선호합니다. TB에서 상위 관리 차원의 동작이 이미 들어가 있거나 들어가야하기 때문입니다. 다소 중복적인 작업이 될 수 있을 것 같습니다.
SP 내부적으로 CSP api call 수준의 여러 오퍼레이션을 반복하여 목적 상태로 최대한 맞춰주시는 것 보다는
CSP가 리턴하는 문제나 상황을 상위에 명확하게 알려주시는 것이 여러모로 더 유용할 것으로 판단됩니다.

예를 들어, SP가 의존성에 의해 삭제하지 못하는 자원이 있다면,
반복 재시도하여 삭제까지 완료해쥬기 보다는
CSP가 제공하는 에러상황(의존성으로 삭제 불가)를 위로 명확하게 전달하고,
SP도 CSP 상황과 동일하게 DB를 유지할 수 있도록만 하면 될 것 같습니다.

다만, VM 생성과 같이 내부적으로 다양한 설정 단계가 포함되어 있는 경우는, 정리하기는 쉽지 않을 것 같네요. VM 생성도 사실 내부 오퍼레이션별 CSP의 상황이 TB로 잘 전파되고, TB에서 세부 단위류 관리 및 제어할 수 있는 상황이 가장 아이디얼 하긴 합니다.

[yunkon-kim]

Note: This issue was not reproduced in the tests for #209.

It may have been resolved with the SP/TB version upgrade. However, I will leave this issue open for the time being.