external-dns should not delete DNS record when HTTPProxy is not completely invalid

[ajayslele]

If there is incorrect (pointing to non-existing child-proxy) or duplicate (multiple inclusions with same match condition) inclusion on a root-proxy instance, it returns status as invalid. However apart from incorrect/duplicate inclusion, the other inclusions continue to function. In such cases, contour should return status like partially-valid. Other status fields like reason can be used to indicate what is wrong. This will allow integrations like external-dns to distinguish between cases where proxy is completely broken and DNS record for virtualhost fqdn can be removed vs. when only some inclusions/routes are not working and it should continue to maintain DNS record for fqdn so that those inclusions/routes can continue to be used. Right now external-dns removes DNS records for all invalid proxies essentially rendering the entire endpoint useless (see https://github.com/kubernetes-sigs/external-dns/blob/master/source/contour_httpproxy.go#L144).

[sunjayBhatia]

Adding to 1.26 milestone for prioritization discussion

[sunjayBhatia]

Some thoughts from triage:

• in HTTPProxy processing, we need to figure out if a resource has produced some valid config or not (similar to the "Accepted" condition in Gateway API, "syntactically and semantically valid enough to produce some configuration...") to be able to differentiate between "full-valid" and "partially-valid"
• if we overload the top-level CurrentStatus field/Valid DetailedCondition to mean that "partially-valid" HTTPProxies will be set to valid there are a couple implications
  • the terms are overloaded
  • right now Valid means no Errors, we may have to change that and unwind any internal logic that assumes that
  • reduces visibility for operators, HTTPProxies that needed to be changed because they were invalid might not be visible
• if we add a new "Programmed" (name TBD) Condition to reflect partially valid
  • we can assume Programmed w/ no Errors is totally valid, Programmed w/ some Errors is partially valid
  • need to teach integrations/operators about this new API surface

[skriss]

xref. https://github.com/projectcontour/contour/blob/main/design/httpproxy-conflict.md for some previous thinking on this topic, that is partially implemented.

[skriss]

ref. #2019
ref. #2141

[ajayslele]

Another option to address the external-dns integration issue mentioned in description can be to provide an additional flag explicitly for create/delete dns record instruction in the proxy status, which external-dns can look at to make the dns record update decision.

[skriss]

Another option to address the external-dns integration issue mentioned in description can be to provide an additional flag explicitly for create/delete dns record instruction in the proxy status, which external-dns can look at to make the dns record update decision.

This would likely be a simpler option in the short term. A possible approach would be to have external-dns look for an annotation on HTTPProxies that would define whether to create a DNS record even if the proxy is not marked "valid". I've been looking through external-dns and don't see any existing functionality that exactly matches this, but I'll open a discussion with that project to get their input.

[lubronzhan]

Might be fixed by kubernetes-sigs/external-dns@42aaa58 (kubernetes-sigs/external-dns#3978)

[skriss]

Closing this out as it is resolved by the above external-dns PR which will create/keep DNS records for root HTTPProxies regardless of whether they are valid or not.

[christianang]

Ended up trying this myself and it should work as expected. Do note though that it will probably be in the next release of external-dns after v0.14.0. The PR @lubronzhan mentioned above was merged a week or so after v0.14.0 was cut.