BackupItemAction plugin timeouts are not reported in backup results file or error count

[priyansh17]

What steps did you take and what happened:
While investigating backup failures involving plugin timeouts, I observed that when a BackupItemAction times out, the backup CR correctly marks the operation as failed and the phase transitions to a partial failure. However, the backup results file (results.json.gz) and the error count field in the backup metadata do not report these timeout errors—they remain unchanged from the initial backup phase. This makes it hard to correlate plugin failures with the content of the backup tarball and results.

What did you expect to happen:
I expected that if any BackupItemAction times out or fails asynchronously, the error would be captured in the backup results file and increment the error count in the backup metadata, so that users can see a summary of all failures (including those that occurred after the initial backup persistence) in the results tarball and error count.

The following information will help us better understand what's going on:

• The backup CR shows BackupItemOperationsFailed and PartiallyFailed phase as expected.
• Timeout errors are present in the per-operation status, but not in the results file or error count.
• The backup tarball uploaded initially is not updated with these errors when the finalizer runs.

Anything else you would like to add:
This behavior may confuse users who expect all errors to be visible in the backup results file and error summary. Please consider updating the finalizer logic to include these errors in results.json.gz and increment the error count accordingly.

Environment:

• Velero version: latest
• Kubernetes version: latest
• Cloud provider: Azure
• OS: Any

[Lyndon-Li]

The content and format of the content in the backup storage are full managed internally by Velero, there is no guarantee for consistency, compatibility, etc. Therefore, the data in the backup storage should not be used by users directly.

If you are using this for DEV scenarios, could you share details of how you use the content and why it is important in your case?

[priyansh17]

The content and format of the content in the backup storage are full managed internally by Velero, there is no guarantee for consistency, compatibility, etc. Therefore, the data in the backup storage should not be used by users directly.

If you are using this for DEV scenarios, could you share details of how you use the content and why it is important in your case?

We use the backup results file to construct warnings in cases of partial success, such as when one persistent volume times out but the rest complete successfully. Also, we use the other backup files such as VSC Contents or Volume Info for internal use cases to understand and store the different volume snapshots created.

In case of timeouts ideally, we report them as partially successful and in warnings show which were timed out but here if they are not consistent, we will end up messing the current solution.

[Lyndon-Li]

Directly interpreting the data in the backup storage is not a recommended way. I suggest you start to investigate to acquire the information through some formal ways, e.g., Velero CRs and Velero APIs (by including some of Velero packages).
Actually, there are already some downstream products which use Velero are generating their own way by using Velero CRs and exsiting Velero codes to retrieve the information for observability.

There is not an official way yet. You are welcome to raise discussions/issues/PRs for this purpose, the final goal is to generate a set of official APIs to retrieve information for observability.
But for sure, interpreting from the backup storage is not a good choice. I cannot say when we will change the format/version of the backup data files in the backup storage, but there are quite some plans requiring this change, so it is really not guaranteed.

cc @reasonerjt

[priyansh17]

Thank for your inputs @Lyndon-Li
I fully agree that directly interpreting backup storage files is risky and support the move toward metric endpoints for observability. That said, there’s an additional aspect to consider, especially for CSI driver users: the error messaging can be inconsistent when certain operations take longer, such as on Azure Kubernetes clusters with PV2 or Ultra disks.

Our intent with this change is specifically to make CSI plugin results consistent. We’re proposing to append error messages based on asynchronous plugin runs, not re-upload or replace files. Building a new CRD just for this use case isn’t trivial for CSI, so our approach aims to improve consistency without introducing breaking changes.

For context, we moved away from using VSC Contents for this reason and now use volumeInfo instead. Since tarballs and related artifacts are sometimes appended at later stages, a similar flow should help us with error message consistency.

To clarify the scenarios:

• In scenario 1 (instant timeout), error messages are already reflected for customers.
• Scenario 2 (async poll/timeout), the current flow doesn’t work well which is why appending errors is preferred here.

I have drafted a PR addressing this. Please have a look it’s not breaking any existing logic but focuses on ensuring the volume info and results are repopulated in a consistent way for CSI users.
#9402

[Lyndon-Li]

there’s an additional aspect to consider, especially for CSI driver users: the error messaging can be inconsistent when certain operations take longer, such as on Azure Kubernetes clusters with PV2 or Ultra disks.

I agree with this scenario, but the key is users will not retrieve the information from the backup metadata directly. So we don't need to change it in the backup data. This is also true for CSI driver users.

[Lyndon-Li]

I have drafted a PR addressing this. Please have a look it’s not breaking any existing logic but focuses on ensuring the volume info and results are repopulated in a consistent way for CSI users.

Async operation has an existing problem and need to solve it future --- it update existing objects in the backup storage, which cause Velero not able to support object lock.
We must fix it as object lock is essential for ransomeware protection.
So we don't want to increase the gap, so any changes that update the existing objects is not preferred.

[priyansh17]

there’s an additional aspect to consider, especially for CSI driver users: the error messaging can be inconsistent when certain operations take longer, such as on Azure Kubernetes clusters with PV2 or Ultra disks.

I agree with this scenario, but the key is users will not retrieve the information from the backup metadata directly. So we don't need to change it in the backup data. This is also true for CSI driver users.

I have drafted a PR addressing this. Please have a look it’s not breaking any existing logic but focuses on ensuring the volume info and results are repopulated in a consistent way for CSI users.

Async operation has an existing problem and need to solve it future --- it update existing objects in the backup storage, which cause Velero not able to support object lock. We must fix it as object lock is essential for ransomeware protection. So we don't want to increase the gap, so any changes that update the existing objects is not preferred.

Thank you for raising these concerns.
If I put myself in the shoes of an end user, the typical place to check backup results is in the metadata and results files generated by Velero. From their perspective, the CRD status only lets them know if a backup was successful, partially failed, or failed, but for understanding exactly what went wrong (such as which resources or volumes failed), users rely on the detailed content in the results file.

If there is an alternative way for users to deeply inspect backup failures beyond the metadata and results file, I’m not aware of it. Most documentation and troubleshooting guides point users to backup result files as the main source of truth for diagnosing issues.

Regarding object lock, while I understand its importance for ransomware protection, it’s currently not supported in Velero. In fact, both the operations controller and the backup_finalizer_controller re-upload the tarball (velero-backup.json) today as part of their normal flow. This re-upload mechanism helps maintain a consistent experience for customers by ensuring result files reflect the latest backup state, including any errors captured post-backup.

Given our current constraints:
object lock limitations, the existing re-upload behavior, and the need for clear error reporting I believe allowing this approach is necessary to deliver a consistent and transparent solution for users.

Are there any other concerns or drawbacks to continuing with re-upload for consistency, apart from the object lock aspect?

[sseago]

I have drafted a PR addressing this. Please have a look it’s not breaking any existing logic but focuses on ensuring the volume info and results are repopulated in a consistent way for CSI users.

Async operation has an existing problem and need to solve it future --- it update existing objects in the backup storage, which cause Velero not able to support object lock. We must fix it as object lock is essential for ransomeware protection. So we don't want to increase the gap, so any changes that update the existing objects is not preferred.

Have you given any thought to how we should fix this? One possibility for the backup tarball is that instead of modifying the tarball during finalize we upload a second "finalize" tarball, and then on restore, we download both tarballs, applying the finalize one second, so if anything is in the finalize tarball that's also in the original tarball, it will overwrite it. This will give the exact same restore behavior as we have now without requiring updates. But I guess we have the backup CR and other files as well. Maybe we need a similar approach everywhere -- a finalize version of each, and the code that downloads these (restore side, backup sync, CLI) will just need to grab the second file (if it exists) and combine them.

[priyansh17]

I have drafted a PR addressing this. Please have a look it’s not breaking any existing logic but focuses on ensuring the volume info and results are repopulated in a consistent way for CSI users.

Async operation has an existing problem and need to solve it future --- it update existing objects in the backup storage, which cause Velero not able to support object lock. We must fix it as object lock is essential for ransomeware protection. So we don't want to increase the gap, so any changes that update the existing objects is not preferred.

Have you given any thought to how we should fix this? One possibility for the backup tarball is that instead of modifying the tarball during finalize we upload a second "finalize" tarball, and then on restore, we download both tarballs, applying the finalize one second, so if anything is in the finalize tarball that's also in the original tarball, it will overwrite it. This will give the exact same restore behavior as we have now without requiring updates. But I guess we have the backup CR and other files as well. Maybe we need a similar approach everywhere -- a finalize version of each, and the code that downloads these (restore side, backup sync, CLI) will just need to grab the second file (if it exists) and combine them.

Thanks for sharing this idea! I think having a second, "finalize" version of the backup tarball avoids in-place modification and keeps the original backup immutable However, for async jobs (especially with BackupItemAction plugins), this approach may result in multiple uploads of the finalize tarball, not just a single second upload. Each time an asynchronous BIA times out or an operation fails late in the process, we'd need to append the new errors or results, resulting in a third, fourth, etc., finalize file version or frequent overwrites.

All components (restore flow, sync controllers, CLI tooling) will need logic to fetch and merge all finalize artifacts, rather than trusting a single, consolidated result set.

[priyansh17]

Artifact (key)	Initial write (backup_controller)	Incremental updates (operations)	Final rewrite (finalizer)
velero-backup.json	YES (PutBackup)	YES (PutBackupMetadata)	YES (PutBackupMetadata)
.tar.gz	YES (PutBackup)	NO	MAYBE (only if finalize produced new tarball; PutBackupContents)
-logs.gz	YES (best effort)	NO	NO
-results.gz	YES (PutBackup)	NO (gap)	NO (not touched)
-itemoperations.json.gz	YES (PutBackup) (initial snapshot)	YES (PutBackupItemOperations)	NO
-volumesnapshots.json.gz	YES	NO	NO
-csi-volumesnapshots*.json.gz	YES	NO	NO
-resource-list.json.gz	YES	NO	NO
-podvolumebackups.json.gz	YES	NO	NO
-volumeinfo.json.gz	YES	NO	NO

These are the current files being updated. I want to suggest updating VolumeInfo and Results