Revise GEP-1713: Update gateway status and conformance test details for ListenerSets

[davidjumani]

What type of PR is this?
Add one of the following kinds:
/kind documentation
/kind gep

Optionally add one or more of the following kinds if applicable:
/area conformance-test

What this PR does / why we need it:
This PR does the following :

• Adds the list of conformance tests to be added to ListenerSets (GEP-1713)
• Updates the status of the Gateway to include AttachedListeners that is the count of successful ListenerSet attachments to the gateway. Ref: slack theraad

The aim is to get consensus for the set of conformance tests required to validate this feature against implementations, and move towards promoting this to Standard

Does this PR introduce a user-facing change?:

Adds the `AttachedListeners` conditions to the Gateway status for the GEP and details for ListenerSets conformance tests

[davidjumani]

cc @rikatz @dprotaso

[robscott]

cc @rostislavbobo @mkosieradzki

[rikatz]

@davidjumani thanks, this is a great write of conformance. Do you think it makes sense to also add the expected conditions on each conformance item, and also the ancestorStatus expected, to make it clear to implementations what we will look for?

[davidjumani]

Do you think it makes sense to also add the expected conditions on each conformance item

I intentionally left them out right now in case there's any discussion to be had when the conformance tests are in review. Happy to add them now if that would clarify things and make reviewing the tests easier.

Added the expected status as well as added the AttachedListeners status that is the count of successful ListenerSet attachments to the gateway. Ref: slack theraad

[rikatz]

should this accepted here be false?

[davidjumani]

Based on the GEP

	// This can be the reason when "Accepted" is "True" or "False", depending on whether
	// the listener being invalid causes the entire Gateway to not be accepted.

I believe this should be true since the ListenerSet is configured but does not adversely affect the parent Gateway.
Additionally, there can be a scenario where only a subset of the listeners is invalid and this would more accurately reflect it

[rikatz]

but then the reason should be empty or something else, right? It was accepted but not programmed, I guess

[rostislavbobo]

+1. If the status is Accepted, there should not be a reason for that.

I wouldn't also mix Accepted: True and Programmed: False in this case. The Programmed condition indicates whether the controller was physically able to apply the requested resources (e.g. allocate an IP or open a port). Listener conflicts are configuration errors, not the controller ability to program them.

[davidjumani]

I don't think so. A subset of listeners can be accepted, so the status should reflect it accordingly.
If not all listeners are accepted, then the listenerSet Accepted condition should be false.

Additionally, the Programmed condition should also reflect the same, since a subset of listeners was applied

[rikatz]

I think there's some confusion here, let me know if the below makes sense

• If I have one listener conflicted:
  • This listener should be accepted (means a controller SAW it), no Reason set
  • This listener should have a status Conflicted=True, Reason explaining if it is a Protocol conflict, etc (and a good message)
  • Programmed will be False, because the proxy won't receive this configuration

As the status conditions is per listener, I think this would make sense, wdyt?

[rikatz]

(specifically on this case, we can think about extending the valid Reasons to something like ConflictWithParent to let the user know there's already a conflicting listener on the Gateway, without leaking any kind of information)

[youngnick]

I think the language in this part is a little confusing:

	// This can be the reason when "Accepted" is "True" or "False", depending on whether
	// the listener being invalid causes the entire Gateway to not be accepted.

I think it should be something like:

	// This can be used as a reason when "Accepted" is "True" or "False", because Accepted
    // requires that at least one Listener present on the object is valid. If zero Listeners in the
    // ListenerSet are valid, then Accepted is False, and this reason is used to explain why.
    // If at least one Listener in the ListenerSet is valid, then Accepted is True, and this reason
    // is used to indicate that the ListenerSet is only _partially_ valid.

Acceptance is about being locally correct, where Programmed is about full correctness. Conflicted is then used to indicate that at least one Listener in the ListenerSet is in conflict with a Listener on another object (no matter which one it is).

I don't think that saying that the conflicted Listener is on the Gateway gives too much information away, since the ListenerSet already includes a reference to that parent anyway. That said, adding a ConflictWithParent reason sounds good to me. It's okay to have more options for Reasons, they don't have the overhead that adding extra whole Conditions does.

[rostislavbobo]

@davidjumani , please add two conformance tests to capture the comment above. Currently, all conflict tests result in Accepted ListenerSets because they have the // Other accepted listeners if applicable comment. We need to test ListenerSets containing only a single conflicting listener:

1. The only listener ls-conflicted on the ListenerSet has a X conflict with a listener gw on the Gateway
   • Gateway status will have AttachedListenerSets: 0
   • ListenerSet will have Accepted: False
2. The only listener ls-conflicted on a ListenerSet ls-conflicted has a X conflict with a listener ls-accepted on another ListenerSet ls-accepted
   • Gateway status will have AttachedListenerSets: 1
   • ListenerSet ls-accepted will have Accepted: True
   • ListenerSet ls-conflicted will have Accepted: False

[davidjumani]

Thanks @rostislavbobo I've added these cases

[rikatz]

do we really need this condition on the parent gateway? I am wondering if in Gateway case we can just not have any count of attached listener, as mostly the user will have its own condition

[davidjumani]

I think the condition should be set since :

• The user has explicitly disallowed ListenerSets
• ListenerSets have been attached (at least tried) to the Gateway

The user should know about this in the status

[rikatz]

I am not aware of the whole conditions reasoning here tbh, will defer to @youngnick for some better view on it

[rikatz]

I am still not fully convinced we should have the condition if we would have a counter as well, but I also do not have a very strong opinion on it

[youngnick]

In this case, I think it's useful for the reason that @davidjumani said - the Gateway owner has said "no ListenerSets allowed", and some ListenerSets have attempted to attach. Regardless of exactly how that situation has come up, the Gateway owner needs to know. attachedListeners doesn't tell us that, because it's a count of ListenerSets that have successfully attached.

[rikatz]

again, should we have an accepted here? And a reason? Not sure if this doesn't get more confusing

[davidjumani]

I think this is a better indication that the listenerSet is correct (the parentref is valid, etc.) but one or more of the listeners is invalid which is indicated by the ListnerStatus above

[rostislavbobo]

but one or more of the listeners is invalid which is indicated by the ListnerStatus above

If all listeners in the ListenerSet have conflicts, shouldn't we report the ListenerSet as not accepted? This is consistent with other resources (e.g. if an HTTPRoute has no hostname intersection with a Gateway listener, it's reported a not accepted).

[davidjumani]

Yes, I agree and will update it accordingly. I'll make this clearer with the total number of conflicting listeners

[rikatz]

@davidjumani overall lgtm! I would like us to keep separated a PR that changes the GEP from a PR that changes the API, this way if we need to track any (or revert) we have proper changes

Thanks again for the effort here!

[rikatz]

do we want to mark this feature as well on supported features? like a Gateway SHOULD report if it supports dynamic port assignment using the feature name "ListenerSetDynamic" (be more creative than I am with names!)

[davidjumani]

Is there an existing feature name for dynamic port assignment on Gateway? If so we should use that - since ListenerSets are just an extension of the gateway

[youngnick]

There is currently no dynamic port assignment on Gateway, port is a required field there. We are trying out dynamic port assignment on ListenerSet.

[davidjumani]

Added the ListenerSetDynamicPortAssignment supported feature flag

[rikatz]

IIRC ReferenceGrant is also a specific supported feature (

gateway-api/pkg/features/referencegrant.go

Line 27 in a863bd3

SupportReferenceGrant FeatureName = "ReferenceGrant"

) so the test needs to be sure that it only happens when ReferenceGrant is supported

[davidjumani]

Agreed. Will add this as a requirement on the specific test

[rikatz]

not sure if this is me being too cautious, but I would like for every test if possible that we also make a request and guarantee that it is working as desired.

eg.: on a conflicting ListenerSet we need to be sure the request goes to the right route attached to the right LS on the right namespace.

Also, optionally, do we want a test on a ListenerSet that has 2 different listeners (eg.: http and https) just to be sure that everything is working fine on some more complex scenarios?

[davidjumani]

Yes, we can do that, and I intend to do so.
I removed it from the doc for now, as I see that not all Gateway conformance tests send out requests but only check the status and I wasn't sure if we wanted to just do it that way.

[rikatz]

/cc
/assign

[rikatz]

Just did a pair review and update with David, I think this is good to go.

/lgtm
/approve

Will wait @youngnick for a second pass, but we can work on conformance based on this already

Thanks!

[youngnick]

Is it useful in this case to record that the Listener is actually Conflicted? (It is, it's just the winner in the conflict stakes).

That would allow Gateway owners to tell that, for example, ListenerSet owners are attempting to configure a Listener that conflicts, and that removing this Listener will allow the ListenerSet Listener to win instead.

This would have to apply to all conflicts between Listeners, including between sibling ListenerSets to work though.

[kflynn]

I feel like it probably is worth marking the Listener as conflicted.

More broadly, I think that it's a good idea to start thinking of a Listener as a sort of degenerate LIstenerSet in basically all cases.

[davidjumani]

I'm not in favour of marking the winner but am open to change. Since gateway listeners have validation that prevents conflicts, there's no precedent on how to handle this. My concerns are :

• A conflicted=true but accepted=true might be confusing to the user
• ListenerSet owners will know that their listener conflicts with another listenerSet and has won. IMO, Ideally they should be isolated and not know about a conflict unless it is not accepted
• If the error message contains the conflicted listener, it might reveal information not privy to the winner

[youngnick]

Thinking more on this, this is probably a thing we can add later if we find we need it. Let's get this in first as is and see.

[youngnick]

Doesn't need to be resolved now, but we will need to come back to what constitutes a "unique" port. If the Protocol is HTTPS, does this port need to be unique if the hostname is distinct? (That would substantially complicate implementing this, and may be a bad idea. But if we are ruling it out, we need to be explicit about that).

[rikatz]

IMO the idea of dynamic would be to really allocate a port for that listener regardless of something similar existing. This would make things easier, and I would say that the dynamic allocation will have more to do in the future with TCP/UDP routes than with HTTPS/TLS routes.

[davidjumani]

I agree that this is more for TCP/UDP routes, however I don't want to rule out the value of this. Shall we write the tests without HTTPS while we get more info on the feasibility of this from implementations?

[youngnick]

No, I think it's probably better to move ahead with HTTPS and see what happens. We have long enough before release that this is fixable if we change our minds.

[rostislavbobo]

Yes, we can do it for HTTPS. However, I’d keep the dynamically assigned port a purely unique for simplicity (without checking hostname fields and crafting a distinct listener because the port number is in used already). If we need this feature, we can always add it later.

[youngnick]

The list of conformance tests looks great to me, as long as we are can answer the two questions I added there.

[rostislavbobo]

Thanks @davidjumani, amazing job! And it was a nice exercise for us all. Please take a look at my comment here: #4205 (comment) . Other than that, it looks good to me

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: davidjumani, rikatz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• geps/OWNERS [rikatz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rikatz]

/lgtm
/hold cancel

This is great, thanks for sticking on it @davidjumani

I am finalizing the conformance test code review as well, but intention is to merge with the current set of tests and then I have already given a heads up to other maintainers you will be fixing the conformance tests during your implementation + reviewing this document