native (brand=joyent) zone in state=incomplete: vmadm delete hangs

[mounty1]

SmartOS 20250918T000550Z

[root@cortex ~]# vmadm get ecc5d57a-3d90-44ca-a89b-8869a120487d
{
"zonename": "ecc5d57a-3d90-44ca-a89b-8869a120487d",
"autoboot": false,
"brand": "joyent",
"limit_priv": "default",
"v": 1,
"create_timestamp": "2025-11-08T09:38:33.162Z",
"image_uuid": "7c4e57a9-1273-40e2-8fcc-569fbbaf6d53",
"cpu_shares": 100,
"max_lwps": 2000,
"max_msg_ids": 4096,
"max_sem_ids": 4096,
"max_shm_ids": 4096,
"max_shm_memory": 1024,
"zfs_io_priority": 100,
"max_physical_memory": 1024,
"max_locked_memory": 1024,
"max_swap": 1024,
"billing_id": "00000000-0000-0000-0000-000000000000",
"owner_uuid": "00000000-0000-0000-0000-000000000000",
"tmpfs": 1024,
"hostname": "imap",
"dns_domain": "landcroft.net",
"resolvers": [
"192.168.1.254"
],
"alias": "IMAP",
"nics": [
{
"interface": "net0",
"mac": "92:8c:af:2b:32:e1",
"nic_tag": "admin",
"ip": "dhcp",
"ips": [
"dhcp"
],
"primary": true
}
],
"datasets": [
"zones/ecc5d57a-3d90-44ca-a89b-8869a120487d/data"
],
"uuid": "ecc5d57a-3d90-44ca-a89b-8869a120487d",
"zone_state": "incomplete",
"zonepath": "/zones/ecc5d57a-3d90-44ca-a89b-8869a120487d",
"hvm": false,
"zoneid": null,
"zonedid": 25,
"last_modified": "2025-11-20T12:26:59.000Z",
"exit_status": 0,
"exit_timestamp": "2025-11-12T10:14:01.203Z",
"firewall_enabled": false,
"server_uuid": "37353236-3837-434e-3736-30373030475a",
"platform_buildstamp": "20250918T000550Z",
"state": "incomplete",
"customer_metadata": {
"root_authorized_keys": "ssh-rsa blah blah blah"
},
"internal_metadata": {},
"routes": {},
"tags": {},
"quota": 4,
"zfs_root_recsize": 131072,
"zfs_filesystem": "zones/ecc5d57a-3d90-44ca-a89b-8869a120487d",
"zpool": "zones",
"zfs_data_recsize": 131072,
"snapshots": []
}

The command vmadm delete ecc5d57a-3d90-44ca-a89b-8869a120487d -f (or without -f) hangs indefinitely.

There is another running instance based on the same image UUID=7c4e57a9-1273-40e2-8fcc-569fbbaf6d53 pkgbuild-lts 24.4.1

The ZFS filesystems under /zones still exist.

The host has been rebooted (several times) since the instance went to stage incomplete.

I can't think what other information might be relevant.

[nshalman]

Do you know/recall how you got into this state in the first place? Did a vmadm create get interrupted? Was this a totally working zone that somehow got corrupted? It's certainly possible to just manually clean up after this mess, but I would really like to understand how we got into this state.

There should be log files under /var/log/vm/logs.
For example, I just created a new VM on my machine and now I have:

# ls -l /var/log/vm/logs/
total 23
-rw-r--r--   1 root     staff       8974 Nov 20 13:08 1763138256166-037406-vmadmd.log
-rw-r--r--   1 root     root       58641 Nov 20 13:08 1763644104429-012350-vmadm.log

Hopefully there should be some further hints as to what is going on inside those log files.

[danmcd]

What does vmadm list and zoneadm list -cv have to say? Another place to look (and maybe upload for sharing) for logs is /zones/$UUID/logs/, especially platform.log.

[mounty1]

The zone was a running zone which I wanted to delete. I cannot now remember whether it was running or stopped when I first ran vmadm delete.

There are no files under /var/log/vm/logs on the host. It is an empty directory.

[root@cortex ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
c8f60c7a-4ce2-49b4-a5d9-8272c74f97b3  OS    768      running           IMAP
ecc5d57a-3d90-44ca-a89b-8869a120487d  OS    1024     incomplete        IMAP
216adb8c-068e-4dfb-8978-11f9700c6ce7  BHYV  4096     running           services
2bc22c01-8d16-4cce-a0fc-f578c77ac5c5  BHYV  4096     running           myCamp
42df2211-5e8d-4e7c-8836-aa0430fe42df  OS    4096     running           Trac
56ffa09a-687b-434a-b6be-7e223c1cb18d  BHYV  4096     running           accounts
20f5fdd6-fc9b-4ede-9695-65ffe1476751  BHYV  8192     stopped           Windows 10
31670940-cb37-453e-ae0a-ba7179f633ec  BHYV  16384    running           Proxmox
423dc3f7-5ddc-4d47-9c4b-7331c9b3e5c9  OS    16384    running           packages

[root@cortex ~]# zoneadm list -cv
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              joyent   shared
   - 20f5fdd6-fc9b-4ede-9695-65ffe1476751 installed  /zones/20f5fdd6-fc9b-4ede-9695-65ffe1476751 bhyve    excl  
  10 31670940-cb37-453e-ae0a-ba7179f633ec running    /zones/31670940-cb37-453e-ae0a-ba7179f633ec bhyve    excl  
   8 216adb8c-068e-4dfb-8978-11f9700c6ce7 running    /zones/216adb8c-068e-4dfb-8978-11f9700c6ce7 bhyve    excl  
   6 423dc3f7-5ddc-4d47-9c4b-7331c9b3e5c9 running    /zones/423dc3f7-5ddc-4d47-9c4b-7331c9b3e5c9 joyent   excl  
   9 2bc22c01-8d16-4cce-a0fc-f578c77ac5c5 running    /zones/2bc22c01-8d16-4cce-a0fc-f578c77ac5c5 bhyve    excl  
   3 42df2211-5e8d-4e7c-8836-aa0430fe42df running    /zones/42df2211-5e8d-4e7c-8836-aa0430fe42df joyent   excl  
   - ecc5d57a-3d90-44ca-a89b-8869a120487d incomplete /zones/ecc5d57a-3d90-44ca-a89b-8869a120487d joyent   excl  
   4 c8f60c7a-4ce2-49b4-a5d9-8272c74f97b3 running    /zones/c8f60c7a-4ce2-49b4-a5d9-8272c74f97b3 joyent   excl  
  11 56ffa09a-687b-434a-b6be-7e223c1cb18d running    /zones/56ffa09a-687b-434a-b6be-7e223c1cb18d bhyve    excl

/zones/ecc5d57a-3d90-44ca-a89b-8869a120487d/logs contains platform.log and console.log, which are attached.

platform.log

console.log

[mounty1]

This is partly user error.

[root@cortex ~]# vmadm update ecc5d57a-3d90-44ca-a89b-8869a120487d indestructible_delegated=false
Successfully updated VM ecc5d57a-3d90-44ca-a89b-8869a120487d
[root@cortex ~]# vmadm delete ecc5d57a-3d90-44ca-a89b-8869a120487d 
Successfully deleted VM ecc5d57a-3d90-44ca-a89b-8869a120487d

but am I being a sore loser if I assert that the behaviour of SmartOS in this situation is sub-optimal?

I.e.: if the zone has an indestructible delegated data-set, say so rather than just hanging.

[nshalman]

but am I being a sore loser if I assert that the behaviour of SmartOS in this situation is sub-optimal?

You are not. I agree that if that property is set we should detect that much earlier and stop with an explanation and leaving the zone exactly as it was when the command was invoked.

[nshalman]

uhhhhhhh..... Now I am very confused as it works the way I would expect for me:

[root@smartos /opt/vmconfigs]# cat delegation_deletion.json
{
  "alias": "delegation_deletion",
  "hostname": "delegation_deletion",
  "autoboot": "false",
  "brand": "joyent",
  "max_physical_memory": 128,
  "image_uuid": "2f1dc911-6401-4fa4-8e9d-67ea2e39c271",
  "quota": 10,
  "delegate_dataset": true,
  "indestructible_delegated": true
}
[root@smartos /opt/vmconfigs]# vmadm create -f delegation_deletion.json
Successfully created VM 69a5c9cb-231e-40e5-979d-28db14ef6505
[root@smartos /opt/vmconfigs]# vmadm delete 69a5c9cb-231e-40e5-979d-28db14ef6505
Failed to delete VM 69a5c9cb-231e-40e5-979d-28db14ef6505: indestructible_delegated is set, cannot delete
[root@smartos /opt/vmconfigs]# vmadm update 69a5c9cb-231e-40e5-979d-28db14ef6505 indestructible_delegated=false
Successfully updated VM 69a5c9cb-231e-40e5-979d-28db14ef6505
[root@smartos /opt/vmconfigs]# vmadm delete 69a5c9cb-231e-40e5-979d-28db14ef6505
Successfully deleted VM 69a5c9cb-231e-40e5-979d-28db14ef6505

Can you reproduce that failure on your machine? If so this is a recently fixed issue?!?!
I don't see any recent changes that would explain this...
Now I am even more confused.

[mounty1]

The crucial difference is that you did not start your instance. If I follow your example, I get your result. If I start the created instance, I get the hang-on-delete behaviour, even if I stop the instance before trying to delete.