Unable to gracefully handle deprecated APIs with Kubernetes manifest

[iamasmith]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.12.2
Kubernetes provider version: 2.38.0
Kubernetes version: 1.32.4

Affected Resource(s)

kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "example" {
  manifest = {
    "apiVersion" = "flowcontrol.apiserver.k8s.io/v1beta3"
    "kind" = "PriorityLevelConfiguration"
    "metadata" = {
      "name" = "example"
    }
    "spec" = {
      "type" = "Limited"
      "limited" = {
        "nominalConcurrencyShares" = 10
        "lendablePercent" = 50
        "limitResponse" = {
          "type" = "Queue"
          "queuing" = {
            "handSize" = 6
            "queueLengthLimit" = 50
            "queues" = 1024
          }
        }
      }
    }
  }
}

Debug Output

Panic Output

│ The plugin returned an unexpected error from
│ plugin.(*GRPCProvider).UpgradeResourceState: rpc error: code = Unknown desc
│ = failed to determine resource GVK: cannot select exact GV from REST mapper

Steps to Reproduce

1. Create a cluster with Kubernetes earlier than 1.32 say 1.28 to be sure that the v1 API is not already present.
2. Apply the resourrce using the v1beta3 API version.
3. Upgrade the cluster to 1.32 without reapplying the TF in between - this version removes this API.
4. Try terraform plan now.

Note now that a Terraform plan of this resource will now fail because the API endpoint is removed.

Changing the API version to v1 from v1beta3 also results in an apply attempting to remove the resource using the old API version and add it back using the new version.

Expected Behavior

What should have happened?

This is debatable and may require a feature to migrate API versions for this kind of resource so that small sections of the state for the resource can be updated to reflect new API versions.

Actual Behavior

What actually happened?

Forced to remove the resource from the cluster, remove the state entry, update the API and reapply.

Important Factoids

https://kubernetes.io/docs/reference/using-api/deprecation-guide/

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[iamasmith]

I took a look through manifest/provider/upgrade_state.go and manifest/morph/morph.go and see there must be some awareness of how resources are handled when they are eventually dropped. I might have a go at creating a couple of dummy CRD versions to simplify the reproduction of this. I'm also curious to see what happens in a little more detail if the old and new API version are available at the point that a subsequent plan occurs. Going from 1.28 to 1.32 as in my example there would clearly be no source API version (v1alpha3 in my example) when 1.32 was active. A CRD will give me flexibility to contrive the test scenarios.
Clearly the behaviour is around the response to the available OpenAPI paths/spiffies available on the kubernetes endpoint in the cluster so a CRD based test will remove the requirement to test around the Kubernetes API itself and allow the test without a cluster upgrade.

[alexsomesan]

@iamasmith Thanks for raising this. Definitely an interesting edge case.

Unfortunately, once a resource is deprecated it gets removed entirely from the API's spec, so there is no way to tell between this case and one where the resource's name and version were misspelled. If the resource is already present in state, we have some additional information but we still cannot continue to manage that resource since the API won't accept requests for it anymore. On the other hand, we cannot trigger a resource deletion during state upgrade phase, but we could "invalidate" it by returning an empty value during the Read phase that follows right after.

How would you feel about that UX?

[iamasmith]

@alexsomesan I think it's a good improvement.

The current state requires that one manually removes the resource, or removes it from source and use a removed directive to drop it from state.

The approach of invalidating it doesn't necessarily prohibit dealing with it in code as described but does offer the opportunity to log that the resource couldn't be found via the API and has been dropped. A corresponding code change could then update the API and if necessary use an import block to reattach the resource without the issue of it hanging around in state.

Personally I can't see any downsides to your proposal.

The approach I was thinking of was more akin to adding an attribute to the resource so that one could demote the legacy API to that attribute and set the API details to the newer one then have the code mutate the resource. I realise though this is tenuous approach unless the previously deprecated API is totally compatible with the replacement so I think your approach is better.