API for Off-Cluster Gateways #3894

kflynn · 2025-07-01T17:44:21Z

As with GEP-3793, I'm sure this will be noncontroversial. 😇

/kind gep

NONE

Signed-off-by: Flynn <[email protected]>

k8s-ci-robot · 2025-07-01T17:44:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kflynn
Once this PR has been reviewed and has the lgtm label, please assign robscott for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

geps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Flynn <[email protected]>

howardjohn · 2025-07-01T18:04:43Z

geps/gep-3792/index.md

+   meshed workloads in the cluster, including providing the OCG with the
+   information it needs to authenticate itself as a workload.
+
+2. The API must allow the mesh to be configured to accept requests from the


In mTLS there are 4 auth pieces. Each party provides credentials and verifies the peer certificates. We seem to only mention "OCG to provide credentials" and "Mesh to verify OCG".

We should require OCG to verify the workload (and the mesh to provide a credential, though this is probably obvious)

Yeah, this a good point: I'm at the point where I think of verifying the peer identity as a required piece of mTLS, but of course it really isn't, and I'll add that as a requirement.

howardjohn · 2025-07-01T18:06:20Z

geps/gep-3792/index.md

+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here
+    trustBundle:


I want to put in consideration adoption of ClusterTrustBundle which is now Beta in Kubernetes. It was designed exactly for this goal and is a perfect fit.

This isn't a "no", but... what would that look like? do we basically vendor the Go type for ClusterTrustBundle for codegen and hope that it doesn't change out from under us? 🤔

No, just an object ref to a ClusterTrustBundle. Or maybe not even a 'ref' needed, just a name (its cluster scoped so no namespace needed)

ClusterTrustBundle, if I understand correctly, is an alpha feature from K8s 1.27 to K8s 1.32, which is to say that in nearly every version supported by Gateway API, if you don't have the feature gate enabled, the ClusterTrustBundle resource type won't exist and there will be no way to refer to a ClusterTrustBundle resource in the Gateway resource... which is why I'm asking what using ClusterTrustBundle would look like. 🙂

(I really, really do not want Gateway API to ship definitions for resources defined by the Kubernetes core, ever.)

ClusterTrustBundle is also a cluster-scoped resource, which feels a bit less than ideal, but that's a much much smaller problem in my mind.

This is also an alpha (experimental) feature. It seems problematic that an experimental feature cannot depend on a Beta feature, and instead we are just re-inventing the exact thing ClusterTrustBundle was designed for

Yeah, this isn't a question about semantics, I kind of agree with you in principle. 🙂 I'm asking about mechanics here: how, technically, can we use ClusterTrustBundle on a cluster where the APIserver does not have any resource type defined under that name?

Saying "you have use K8s 1.33 or enable the feature gate" feels problematic.

The options here are:

we only add trustBundle inline for now, with a note about how we will add ClusterTrustBundle in the future.

we add both for now, with a note that once ClusterTrustBundle is more available in Gateway API supported versions, we will remove the inline trustBundle. We can get away with that as long as this resource is Experimental.

We only add trustBundle, and say "run 1.33 or later, or enable an alpha feature to use this". This would require us to specify the behavior for implementations; since this has a FeatureName anyway, there is a mechanism to say "requires k8s 1.33 or alpha feature enabled to work", although we've never done that before.

Because Gateway API's claimed Kubernetes support is five versions, the last would mean that we would need to wait nearly two years before bringing this API to GA in options 2 or 3. That feels like a longer timeline than people would expect for this. Unless we are okay with doing 3 and adding features that are not supported across all supported Kubernetes versions, which does not seem optimal to me.

@robscott for his thoughts on this one too.

Actually, I just checked, and ClusterTrustBundle is only beta, and not enabled by default in v1.33, so this would be even longer. (See https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#cluster-trust-bundles for the reference)

At a minimum, if ClusterTrustBundle moves to Stable in v1.34, then the only time we could allow it with no caveats is as of Kubernetes v1.38, which is in just over two years.

In short; my thoughts here are that we can add a ClusterTrustBundle reference with a feature name and a version caveat at best. But we need something for before then as well.

We have similar (though less restrictive) versioning issues for using ClusterTrustBundle within GKE. The approach we took was to make a CustomResource ("GKEClusterTrustBundle") that had exactly the same shape as ClusterTrustBundle, and have our controllers understand both types.

howardjohn · 2025-07-01T18:14:53Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


This applies to the whole proposal in general but specifically this line. There are 3 OCG implementations today. I would think to make this viable we need at least 2 to agree to this.

I know that, unless things have changed, using a custom ALPN is the hill at least one cloud will die on and will never support

I'm not comfortable making this a MUST. Custom ALPNs are certainly a way to signal mTLS, but different implementations may use tunneling protocols like HBONE instead.

@keithmattix and I discussed this on Slack a bit, since I'm very uncomfortable with making this a MAY and then having, say, the OCG expecting ALPN, the mesh expecting HBONE, but they both believe that they "support GEP-3792 OCG".

As Keith noted on Slack, though, it doesn't really make sense to require a mesh to only support one mechanism for solving the protocol problem, and it follows that it probably does make sense to add an explicit field to the Gateway and Mesh resources to say what the signalling mechanism is. That also permits warning Chihiro if the Gateway and Mesh disagree on what's offered. I'll add that.

Also, to @howardjohn's point: we explicitly say in "Graduation Criteria" section that we require at least two OCG implementations and at least two mesh implementations. Leaving aside the choice of hills upon which to die... 🙂

keithmattix · 2025-07-01T18:56:53Z

geps/gep-3792/index.md

+  that the OCG should trust when communicating with meshed workloads in the
+  cluster.
+
+- The trust bundle in the Mesh resource will define the CA certificate(s)


I may have missed this in another GEP, but who's creating the Mesh resource: the user or the mesh controller? How does this get plumbed through? IIRC, this ambiguity was one of the reasons we didn't originally create a mesh resource

I was kinda hoping that maybe someone else would write the Mesh-resource GEP. 😉 It probably makes sense to include it in this one, though.

My thought here is that Chihiro creates the Mesh resource, and that the Mesh resource includes a controllerName field that identifies its owner. (I don't think that a MeshClass resource makes sense now.) If Chihiro doesn't create the Mesh resource, then the mesh won't enable OCG (because it won't have the trust bundle, etc.).

keithmattix · 2025-07-01T19:00:07Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


I'm not comfortable making this a MUST. Custom ALPNs are certainly a way to signal mTLS, but different implementations may use tunneling protocols like HBONE instead.

keithmattix · 2025-07-01T19:05:09Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


One other option that randomly popped in my head (not saying it's a great one): add some metadata to backendRef. This shouldn't at all take any of backendTLSPolicy's scope, but it's a way for the route to say "this backend is meshed"

I would really like to arrange things so that Ana needn't take on that burden. But let's keep it in mind.

keithmattix · 2025-07-01T19:06:21Z

geps/gep-3792/index.md

+Gateway resource, and to create a Mesh resource which will also have a trust
+bundle:
+
+- The trust bundle in the Gateway resource will define the CA certificate(s)


~~A single trust bundle implies that multitenancy (i.e. multiple meshes with different trust domains served by the same gateway) is a non-goal for now?~~

I see below that it's a list of secrets; should this be pluralized?

"Trust bundle" to mean "one or more CA certificates that you're going to trust" is pretty well-established in the PKI world. Pluralizing it would imply having multiple lists, rather than multiple certificates; I don't really think that that makes sense.

Oh wait I was interpreting this in light of John's comment about using CTB. In which case it got sort of confusing

To respond to your overstruck comment: you are correct, support for multiple meshes in the same cluster is a non-goal in this pass. I'll make that explicit as well.

geps/gep-3792/index.md

keithmattix · 2025-07-01T19:08:48Z

geps/gep-3792/index.md

+
+```yaml
+apiVersion: networking.x-k8s.io/v1
+kind: Mesh


s/Mesh/XMesh?

Uhhhhhhhhh wow, that's a great question. Yeah, probably so.

geps/gep-3792/index.md

youngnick · 2025-07-02T03:14:27Z

I won't get much of a chance to watch this one further, but the main point I wanted to add is that, because this is TLS config for a Gateway, it's important that we define the interaction (or lack thereof) with consumer BackendTLSPolicy as per #3875.

FTR, I think that saying "they are for different things, the only interaction is to end up with TLS-in-TLS, which is dumb, so you should avoid that". is fine by me.

API for GEP-3792

bd16376

Signed-off-by: Flynn <[email protected]>

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/gep PRs related to Gateway Enhancement Proposal(GEP) cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 1, 2025

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 1, 2025

k8s-ci-robot requested review from gcs278 and robscott July 1, 2025 17:44

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 1, 2025

Typo

a1bc550

Signed-off-by: Flynn <[email protected]>

howardjohn reviewed Jul 1, 2025

View reviewed changes

keithmattix reviewed Jul 1, 2025

View reviewed changes

kflynn changed the title ~~Add API for Off-Cluster Gateways~~ API for Off-Cluster Gateways Jul 1, 2025

API for Off-Cluster Gateways #3894

Are you sure you want to change the base?

API for Off-Cluster Gateways #3894

Conversation

kflynn commented Jul 1, 2025

Uh oh!

k8s-ci-robot commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youngnick Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kflynn Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

youngnick commented Jul 2, 2025

Uh oh!

Uh oh!

youngnick Jul 2, 2025 •

edited

Loading

kflynn Jul 1, 2025 •

edited

Loading