Secret silently not restored

[nyarly]

I have a secret that's not restored in my tests of my production cluster. When I create the backup, I see it logged. It appears in the resource list of the backup. When I restore, the secret is logged as being restored, but the restored cluster lacks the secret.

The secret is named dev/vault-auth-tokenreview-token in this example. I suspect that the generated dev/vault-auth-tokenreview-token-j5sql somehow "masks" the secret I need restored; to be clear they are not the same.

⮀ rg -C 3 vault-auth-tokenreview-token backup-logs/test-backup-5*
backup-logs/test-backup-5.describe
    - dev/tier-zero-keycloak-token-w2wtg
    - dev/tier-zero-vault-agent-injector-token-dndls
    - dev/tier-zero-vault-token-4mll4
    - dev/vault-auth-tokenreview-token
    - dev/vault-auth-tokenreview-token-j5sql
    - dev/vault-gsuite
    - dev/vault-poststart
    - dev/vault-server-tls

backup-logs/test-backup-5.log
time="2021-08-17T04:32:17Z" level=info msg="Processing item" backup=management/test-backup-5 logSource="pkg/backup/backup.go:354" name=tier-zero-vault-token-4mll4 namespace=dev progress= resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backing up item" backup=management/test-backup-5 logSource="pkg/backup/item_backupper.go:121" name=tier-zero-vault-token-4mll4 namespace=dev resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backed up 672 items out of an estimated total of 1792 (estimate will change throughout the backup)" backup=management/test-backup-5 logSource="pkg/backup/backup.go:394" name=tier-zero-vault-token-4mll4 namespace=dev progress= resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Processing item" backup=management/test-backup-5 logSource="pkg/backup/backup.go:354" name=vault-auth-tokenreview-token namespace=dev progress= resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backing up item" backup=management/test-backup-5 logSource="pkg/backup/item_backupper.go:121" name=vault-auth-tokenreview-token namespace=dev resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backed up 673 items out of an estimated total of 1792 (estimate will change throughout the backup)" backup=management/test-backup-5 logSource="pkg/backup/backup.go:394" name=vault-auth-tokenreview-token namespace=dev progress= resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Processing item" backup=management/test-backup-5 logSource="pkg/backup/backup.go:354" name=vault-auth-tokenreview-token-j5sql namespace=dev progress= resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backing up item" backup=management/test-backup-5 logSource="pkg/backup/item_backupper.go:121" name=vault-auth-tokenreview-token-j5sql namespace=dev resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backed up 674 items out of an estimated total of 1792 (estimate will change throughout the backup)" backup=management/test-backup-5 logSource="pkg/backup/backup.go:394" name=vault-auth-tokenreview-token-j5sql namespace=dev progress= resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Processing item" backup=management/test-backup-5 logSource="pkg/backup/backup.go:354" name=vault-gsuite namespace=dev progress= resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backing up item" backup=management/test-backup-5 logSource="pkg/backup/item_backupper.go:121" name=vault-gsuite namespace=dev resource=secrets
time="2021-08-17T04:32:17Z" level=info msg="Backed up 675 items out of an estimated total of 1792 (estimate will change throughout the backup)" backup=management/test-backup-5 logSource="pkg/backup/backup.go:394" name=vault-gsuite namespace=dev progress= resource=secrets

backup-logs/test-backup-5-20210817092107.log
time="2021-08-17T16:22:09Z" level=info msg="Restored 163 items out of an estimated total of 1532 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=tier-zero-vault-agent-injector-token-dndls namespace=dev progress= resource=secrets restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Attempting to restore Secret: tier-zero-vault-token-4mll4" logSource="pkg/restore/restore.go:1238" restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Restored 164 items out of an estimated total of 1532 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=tier-zero-vault-token-4mll4 namespace=dev progress= resource=secrets restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Attempting to restore Secret: vault-auth-tokenreview-token-j5sql" logSource="pkg/restore/restore.go:1238" restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Restored 165 items out of an estimated total of 1532 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=vault-auth-tokenreview-token-j5sql namespace=dev progress= resource=secrets restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Attempting to restore Secret: vault-auth-tokenreview-token" logSource="pkg/restore/restore.go:1238" restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Restored 166 items out of an estimated total of 1532 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=vault-auth-tokenreview-token namespace=dev progress= resource=secrets restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Attempting to restore Secret: vault-gsuite" logSource="pkg/restore/restore.go:1238" restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Restored 167 items out of an estimated total of 1532 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:664" name=vault-gsuite namespace=dev progress= resource=secrets restore=management/test-backup-5-20210817092107
time="2021-08-17T16:22:09Z" level=info msg="Attempting to restore Secret: vault-poststart" logSource="pkg/restore/restore.go:1238" restore=management/test-backup-5-20210817092107

[felfa01]

I have seen something similar. I have a use case where I require the Secret token to be restored as it was and not skipped (as described here) for a new token to be generated in its place.

We are providing access to team specific AKS namespaces via Service Accounts. The teams can access their namespaces via a Service Connection in e.g. Azure DevOps by referencing a Secret token generated via Service Account. The issue is that after running a Velero restore, the Secret token is re-created (e.g. new token value) and all our application teams are required to update their Service Connections with the new token. This is not feasible from a UX perspective.

Anyone that has a workaround for this issue?

[kasabov]

I know this is an old story but I just posted something #2524 (comment) and thought to cross link it here.
TL;DR: Store SA (and SA token secrets) in another namespace (e.g. kube-system). Only the rolebinding needs to be in the specific namespace.