kubernetes_manifest to Prefer OpenAPI v3 schema

[yhrn]

Description

Schema lookup for kubernetes_manifest resources currently works as follows:

1. Determine if the resource type is defined by a CRD and if it is extract the schema from the CRD.
2. Otherwise fall back to the /openapi/v2 endpoint (which serves the entire spec as a single, massive, document)

The reason for this approach (based on this comment) is that CRDs have OpenAPI v3 schema so using the v2 schemas served at /openapi/v2 would mean using a "lossy view" of the schema, e.g. field defaults are not superted by v2.

However, in Kubernetes 1.27 (released April 2023) OpenAPI v3 became GA and is served at /openapi/v3. My proposal is to use this as the preferred source for schemas for all resource types, CRD based or not. The current approach could be the fallback for older k8s versions, but I'm not sure how old versions you want to support (1.26 became unsupported in March 2024)?

Benefits of this approach:

• No need to list CRDs, which should have performance and access control benefits as indicated by the issues in the references section.
• Higher fidelity for non CRD types - as stated in the linked Kuberenetes blog post, some built in types are not completely correctly described by the OpenAPI vs spec. It would also make the provider compatible with other k8s derivatives that introduces other mechanisms than CRDs for extending the API (e.g. KCP)
• Potential performance benefits over OpenAPI v2 - since the v3 spec is served as separate documents per API group/version it may require significantly less data to be transferred.
• Cleaner, easier to and understand and maintain (assuming the current behavior can be dropped at some point)

I'm interested in helping out with implementing this but I'd first like to get feedback on the idea and on how to handle older k8s version (v1.26 and down) - no support, fallback to the current behavior, controlled via provider configuration, etc.

Potential Terraform Configuration

N/A (unless the behavior should be user controlled via provider configration)

References

• Kubernetes blog post about OpenAPI v3 support GA
• CRD api access error on kubernetes_manifest prevents a non-CRD resource to be created. #1507
• kubernetes_manifest listing all CRDs each time #1651
• kubernetes_manifest resource should NOT make any CRD call  #1665
• feature: ability to manage CRDs in KCP using terraform kcp-dev/kcp#3333

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment