Backup hangs in Queued status

[Lyndon-Li]

Steps:

• Make BSL unavailable
• Create one backup, test07, with namespace NS-1
• test07 stays in FailValidation phase
• Fix BSL problem and make BSL turns to available
• Create another backup, test08, with the same namespace NS-2
• test08 starts with Queued status
• Delete test07
• test08 is not started and hang in Queued status
• (I waited for more than 10 min)

backups-202601220705.0765.json

velero.log

[sseago]

@Lyndon-Li Good catch. This is definitely a release blocker. It should be a relatively easy fix, though. I'm off tomorrow, but I'll focus on this fix on Monday.

[shubham-pampattiwar]

Traced through the code and hopefully found the root cause.

Problem is in

velero/pkg/controller/backup_controller.go

Lines 310 to 316 in 7442d20

	if request.Status.Phase == velerov1api.BackupPhaseFailedValidation {
	log.Debug("failed to validate backup status")
	b.metrics.RegisterBackupValidationFailure(backupScheduleName)
	b.metrics.RegisterBackupLastStatus(backupScheduleName, metrics.BackupLastStatusFailure)
	return ctrl.Result{}, nil
	}

When a backup moves to ReadyToStart, the queue controller adds it to the tracker via backupTracker.AddReadyToStart(). This increments RunningCount().

When validation fails, the backup controller sets the phase to FailedValidation and returns early at line 315. The issue is that this early return happens before backupTracker.Add() at line 321 and the deferred backupTracker.Delete() setup at lines 322-329

So the backup stays orphaned in the readyToStartBackups set. RunningCount() keeps returning 1, and any subsequent queued backup gets blocked at

velero/pkg/controller/backup_queue_controller.go

Line 274 in 7442d20

if r.backupTracker.RunningCount() >= r.concurrentBackups {

Deleting the failed backup doesn't help either since there's no cleanup path for the tracker entry.

Proposed Fix: Add b.backupTracker.Delete(request.Namespace, request.Name) before the early return at line 315

[sseago]

Yes, that looks like it. I will either implement @shubham-pampattiwar's proposed fix or maybe just move the backupTracker.Add() and defer delete to before any validation. The point of the defer call is to make sure we remove it no matter how the function exits. My bug was in not making that defer call before the first possible return.

[shubham-pampattiwar]

yeah thats cleaner, sounds good to me !

[Lyndon-Li]

Thanks @sseago @shubham-pampattiwar , I am adding this issue to 1.18-rc