Unable to clear Finalizers resulting in the PR/TRs even with deletiontimestamp stuck in the cluster

[anithapriyanatarajan]

Expected Behavior

Once the PipelineRun (PR) is completed, the Tekton Chains controller adds a finalizer to handle provenance creation and artifact signing. After this process is finished and a deletion timestamp is set on the PR, the finalizer will be CLEARED, allowing the object to be deleted.

Actual Behavior

THE FINALIZER CLEAR FAILS with the below error:

error":"failed to clear finalizers: admission webhook \"validation.webhook.pipeline.tekton.dev\" denied the request: validation failed: invalid value: Once the PipelineRun is complete, no updates are allowed:

There is no retry following this resulting in the object stuck until we clear them manually.

Steps to Reproduce the Problem

This issue is not reproducible all the time. But we do have evidence of stuck PRs in production clusters.

[vdemeester]

@anithapriyanatarajan can you share the version of pipeline and chains that were experiencing it ?
It sounds similar to #8230 but it's a different case, so it might not have been fully fixed.

[anithapriyanatarajan]

@anithapriyanatarajan can you share the version of pipeline and chains that were experiencing it ? It sounds similar to #8230 but it's a different case, so it might not have been fully fixed.

This was reported in OpenShift Cluster running Pipelines version 1.71.1 which encompasses Pipelines component version - 0.65.6 & Chains version - 0.23.1