GCP snapshots not deleted after backup TTL expires; no controller:backup-deletion logs emitted #9480
Description
What steps did you take and what happened:
We are running Velero on multiple Kubernetes clusters in GCP (GKE) with GCP plugins and a dedicated Kubernetes ServiceAccount for Velero.
We have a scheduled backup every 24 hours with a TTL of 10 days. Backups are successfully created, and when TTL expires, the backup object and backup data are deleted from the bucket as expected.
However, on some clusters, when TTL expires the GCP native snapshots are not deleted. In the affected clusters, we also do not see any controller:backup-deletion logs at the time the TTL expires and the backup is removed.
There is no known configuration/setting/version difference between clusters where snapshot deletion works and clusters where it fails.
What we observed in problematic clusters
Backup is created normally (daily schedule).
TTL expires, and backup data is removed from object storage.
Snapshots remain in GCP after TTL expiration.
No controller:backup-deletion logs are emitted around the TTL expiration/deletion time.
What did you expect to happen:
When the backup TTL expires and the backup is deleted, we expect Velero to also delete all associated GCP native snapshots created for that backup (as it does in other clusters with the same setup).
The following information will help us better understand what's going on:
This issue occurs only on some clusters. On working clusters (same cloud provider, same schedule/TTL concept), Velero deletes both backup data and the associated GCP snapshots successfully.
In affected clusters, the absence of controller:backup-deletion logs suggests the backup deletion controller may not be running/triggering, or deletion requests are not being processed for some reason—even though the backup content is removed.
Environment:
Velero version (use velero version):
Client: v1.15.1
Server: v1.13.2
Velero features (use velero client config get features):
Kubernetes version (use kubectl version):
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.33.5-gke.1791000
Kubernetes installer & version:
GKE (v1.33.5-gke.1791000)
Cloud provider or hardware configuration:
Google Cloud Platform (GCP), using GCP native snapshots via Velero GCP plugin