Unable to delete Dynamic Host Volume

[shochdoerfer]

Nomad version

v1.10.2 (Nomad Master, Nomad Clients, and my local CLI client)

Issue

Yesterday, I tried to delete a Dynamic Host Volume in 2 of our Nomad clusters, and both failed to delete the volume.

Nomad CLI returned the following error:
Error deleting volume: Unexpected response code: 500 (could not delete volume 7f8d6ba2-992e-556a-5382-ba976c38febd in use by alloc b102ceb2-8fe3-9e8e-d465-d1c84f36502b)

In the Nomad Web UI the storage state got set to: unavailable

According to this PR, the Nomad client now has a --force flag to force the deletion of a volume, but even that had no effect. Before trying the force command, I updated both Nomad clusters to version 1.10.2.

Reproduction steps

I don't think its easily reproducible. I created and deleted various dynamic host volumes in the last few weeks and never run into an issue.

Expected Result

The dynamic host volume should be deleted since no job was depending on it anymore.

Actual Result

The dynamic host volume is not deleted and is now stuck in an unavailable state.

The only way to solve the issue was to drain the node, stop the Nomad client, run the garbage collector, and start the client again. Afterwards, I could delete the volume with the same command as before.

The volume was not in use for several days. Maybe somehow Nomad thought it still is?

Also, what's weird, I could not get the 1.10.2 version of Nomad for my Ubuntu Focal instance. Apt kept saying that my 1.10.1 version is the latest. Is Ubuntu Focal no longer supported?

[pkazmierczak]

Hi @shochdoerfer, thanks for reporting this. What was the host volume plugin that you used?

[shochdoerfer]

I've been using the mkdir plugin.

[pkazmierczak]

Hey @shochdoerfer, I tried reproducing this but no luck. I'll mark it for further investigation.

[tgross]

@shochdoerfer this error message might give us a clue as to what's happening here:

Error deleting volume: Unexpected response code: 500 (could not delete volume 7f8d6ba2-992e-556a-5382-ba976c38febd in use by alloc b102ceb2-8fe3-9e8e-d465-d1c84f36502b)

The only way we should ever get that error is if there are non-client-terminal allocations on that volume's node that have the volume in their task group. Can you run nomad alloc status b102ceb2 and report back what you see? I wonder if the allocation in question is DesiredStatus: stop but ClientStatus: running (or some other non-terminal state).

[shochdoerfer]

@tgross sure, here you go:

ID                  = b102ceb2-8fe3-9e8e-d465-d1c84f36502b
Eval ID             = 87d41947
Name                = thanos.thanos[0]
Node ID             = d3c54071
Node Name           = nomad-client1
Job ID              = thanos
Job Version         = 22
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 3d14h ago
Modified            = 3d14h ago
Deployment ID       = 7979c218
Deployment Health   = healthy

Allocation Addresses:
Label          Dynamic  Address
*querier-grpc  yes      10.10.11.2:10901 -> 10901
*sidecar-grpc  yes      10.10.11.2:31150 -> 10901
*sidecar-http  yes      10.10.11.2:27014 -> 10902
*querier-http  yes      10.10.11.2:28121 -> 10902

Nomad Service Checks:
Service                  Task                Name                                Mode         Status
thanos-sidecar-http-dc1  thanos-sidecar-dc1  thanos-sidecar-http-dc1 port alive  healthiness  success

Task "thanos-compactor-dc1" is "running"
Task Resources:
CPU        Memory          Disk     Addresses
0/100 MHz  70 MiB/300 MiB  300 MiB

Host Volumes:
ID                Read Only
thanos-compactor  false

Task Events:
Started At     = 2025-06-23T16:55:14Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2025-06-23T18:55:14+02:00  Started     Task started by client
2025-06-23T18:55:13+02:00  Task Setup  Building Task Directory
2025-06-23T18:55:13+02:00  Received    Task received by client

Task "thanos-querier-dc1" is "running"
Task Resources:
CPU        Memory          Disk     Addresses
0/100 MHz  43 MiB/300 MiB  300 MiB

Task Events:
Started At     = 2025-06-23T16:55:22Z
Finished At    = N/A
Total Restarts = 1
Last Restart   = 2025-06-23T16:55:22Z

Recent Events:
Time                       Type              Description
2025-06-23T18:55:22+02:00  Started           Task started by client
2025-06-23T18:55:22+02:00  Restarting        Task restarting in 0s
2025-06-23T18:55:22+02:00  Terminated        Exit Code: 0
2025-06-23T18:55:21+02:00  Restart Signaled  Template with change_mode restart re-rendered
2025-06-23T18:55:14+02:00  Started           Task started by client
2025-06-23T18:55:13+02:00  Task Setup        Building Task Directory
2025-06-23T18:55:13+02:00  Received          Task received by client

Task "thanos-sidecar-dc1" is "running"
Task Resources:
CPU        Memory          Disk     Addresses
0/100 MHz  38 MiB/300 MiB  300 MiB

Host Volumes:
ID              Read Only
thanos-sidecar  false

Task Events:
Started At     = 2025-06-23T16:55:13Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2025-06-23T18:55:13+02:00  Started     Task started by client
2025-06-23T18:55:13+02:00  Task Setup  Building Task Directory
2025-06-23T18:55:13+02:00  Received    Task received by client

[tgross]

Ok, yeah. @shochdoerfer the allocation is running and even have Desired Status: run, which means the job has not been stopped. We can't safely delete a volume that's in use by the allocation, so we don't. That's the error message you received.

Are you sure the job thanos is actually stopped? There's nothing here indicating that it is.

[shochdoerfer]

@tgross "It's complicated" :) Well, actually not. The job is running (again) but not using the volume. I moved it over to use a different dynamic host volume. Also, in the volume details, it does not show me any job using the volume.

Will try later what happens if I stop the job and then try to delete the volume.