🚀 Reattach to new destroy run when deleting a workspace

[baptman21]

Description

This is not a bug report per se, because it is the current behavior of the code, but a feature request to improve the current behavior.

Currently, when you have a Workspace, if a run is in errored state, but you start a new one manually, it is capable of getting the new run's status.

Run status	K8S Workspace status
run 572e: running	running
run 572e: errored	errored
run 7342: pending (started manually)	pending
run 7342: running	running
run 7342: applied	applied

However, when you are deleting a workspace, the same is not true if the destroy run fails.

Destory run status	K8S workspace status
run 1290: running (deleting)	running (deleting)
run 1290: errored	errored
run 7554: pending (deleting) (started manually)	errored
run 7554: running (deleting)	errored
run 7554: applied	errored

It would be nice if when you start a new destroy run on the operator was able to reattach to it.

I see 2 possibilities:

1. the operator can fetch the latest destroy run just like it does for the normal runs
2. the operator checks when deleting if it can do a safe delete first. This way if you started a run manually and deleted the resources, the workspace would be deleted and not be stuck in deleting

Logs

The tables above are simplifications but the logs look like the following. When I cancel the run manually (to simulate an error in the delete), and start a new one, the operator is stuck on destroy run run-yAKWfHAC284yJsuZ is unsuccessful: canceled even when I started a new one.

2025-01-20T09:24:02Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "new reconciliation event"}
2025-01-20T09:24:02Z    INFO    Spec Validation {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "validating instance object spec"}
2025-01-20T09:24:02Z    INFO    Spec Validation {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "spec is valid"}
2025-01-20T09:24:02Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "reconciling workspace"}
2025-01-20T09:24:02Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "object marked as deleted, need to delete workspace first"}
2025-01-20T09:24:02Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "deletion policy is destroy"}
2025-01-20T09:24:02Z    INFO    Destroy Run     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "get destroy run run-yAKWfHAC284yJsuZ"}
2025-01-20T09:24:02Z    INFO    Destroy Run     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "destroy run run-yAKWfHAC284yJsuZ is not finished"}
2025-01-20T09:24:02Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "successfully reconcilied workspace"}
2025-01-20T09:24:02Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "current run run-yAKWfHAC284yJsuZ status plan_queued is not completed need to requeue"}

2025-01-20T09:24:31Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "new reconciliation event"}
2025-01-20T09:24:31Z    INFO    Spec Validation {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "validating instance object spec"}
2025-01-20T09:24:31Z    INFO    Spec Validation {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "spec is valid"}
2025-01-20T09:24:31Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "reconciling workspace"}
2025-01-20T09:24:31Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "object marked as deleted, need to delete workspace first"}
2025-01-20T09:24:31Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "deletion policy is destroy"}
2025-01-20T09:24:31Z    INFO    Destroy Run     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "get destroy run run-yAKWfHAC284yJsuZ"}
2025-01-20T09:24:32Z    INFO    Destroy Run     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "destroy run run-yAKWfHAC284yJsuZ is unsuccessful: canceled"}
2025-01-20T09:24:32Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "successfully reconcilied workspace"}
2025-01-20T09:24:32Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "current run run-yAKWfHAC284yJsuZ status plan_queued is not completed need to requeue"}

2025-01-20T09:25:02Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "new reconciliation event"}
2025-01-20T09:25:02Z    INFO    Spec Validation {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "validating instance object spec"}
2025-01-20T09:25:02Z    INFO    Spec Validation {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "spec is valid"}
2025-01-20T09:25:02Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "reconciling workspace"}
2025-01-20T09:25:02Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "object marked as deleted, need to delete workspace first"}
2025-01-20T09:25:02Z    INFO    Reconcile Workspace     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "deletion policy is destroy"}
2025-01-20T09:25:02Z    INFO    Destroy Run     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "get destroy run run-yAKWfHAC284yJsuZ"}
2025-01-20T09:25:02Z    INFO    Destroy Run     {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "destroy run run-yAKWfHAC284yJsuZ is unsuccessful: canceled"}
2025-01-20T09:25:02Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "successfully reconcilied workspace"}
2025-01-20T09:25:02Z    INFO    Workspace Controller    {"workspace": {"name":"my-workspace","namespace":"my-workspace"}, "msg": "current run run-yAKWfHAC284yJsuZ status plan_queued is not completed need to requeue"}

Side note

This would be a nice feature also, as it would simplify the creation of a retry feature, as requested in another issue (#163). If the operator can reattach to the latest delete run, then the retry strategy does not have to interact with the deleting process, but can be as simple as:

• Did the latest run (normal or delete) fail ?
  • If so, start a new one
    And when the new run is started on the next reconciliation, the operator picks up the latest retry.

References

• GH-163

Community Note

• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.