Annotate HTTPRoute when detecting an issue with EndpointSlice + Service configuration #6269
Description
Description:
When routing to a Service backend which owns a EndpointSlice with a broken owner reference, HTTPRoute should be annotated to describe the error. At the moment EnvoyGateway will add a static 500 direct response route in its place, but there is no log message or annotation on resources to convey that there is a problem.
Repro steps:
This is reproducible when configuring a Service backend, which owns an EndpointSlice, but the owner reference label isn't correct. Note the incorrect owner reference label on the EndpointSlice kubernetes.io/service-name: not-the-correct-reference
I was able to reproduce this using these steps:
kind create cluster --name envoy-gateway-500 --wait 300s
kubectl get namespace envoy-gateway-system || kubectl create namespace envoy-gateway-system
helm upgrade eg oci://docker.io/envoyproxy/gateway-helm --version v1.4.0 -n envoy-gateway-system --install
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
kubectl apply -f envoy-resources.yaml
egctl x status all -A
ENVOY_POD=$(kubectl get pods -n envoy-gateway-system -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=envoy-gateway-system,gateway.envoyproxy.io/owning-gateway-name=public -o jsonpath='{.items[0].metadata.name}')
kubectl wait --for=condition=Ready pod/$ENVOY_POD -n envoy-gateway-system
ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=envoy-gateway-system,gateway.envoyproxy.io/owning-gateway-name=public -o jsonpath='{.items[0].metadata.name}')
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &
curl -v -H 'Host: www.example.com' localhost:8888/json
# Will result in a 500 error directly from envoy gatewayenvoy-resources.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: public
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
namespace: envoy-gateway-system
name: public-gateway-config
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: public-gateway-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 1
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: public
namespace: envoy-gateway-system
spec:
gatewayClassName: public
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
hostname: "www.example.com"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin
namespace: envoy-gateway-system
spec:
hostnames:
- "www.example.com"
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: public
namespace: envoy-gateway-system
sectionName: http
rules:
- backendRefs:
- group: gateway.envoyproxy.io
kind: Service
name: httpbin
port: 80
filters:
- type: URLRewrite
urlRewrite:
hostname: httpbingo.org
matches:
- path:
type: PathPrefix
value: /
timeouts:
backendRequest: 5s
request: 5s
---
apiVersion: v1
kind: Service
metadata:
name: httpbin
namespace: envoy-gateway-system
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
name: https
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: httpbin
namespace: envoy-gateway-system
labels:
kubernetes.io/service-name: not-the-correct-reference
addressType: FQDN
ports:
- name: https
protocol: TCP
port: 80
endpoints:
- addresses:
- "httpbingo.org"Environment:
Envoy Gateway 1.4.0
Logs:
Envoy Proxy logs
shutdown-manager 2025-06-06T21:18:22.217Z INFO shutdown-manager envoy/shutdown_manager.go:74 starting shutdown manager
envoy {":authority":"www.example.com","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:48212","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"direct_response","response_flags":"-","route_name":"httproute/envoy-gateway-system/httpbin/rule/0/match/0/www_example_com","start_time":"2025-06-06T21:19:16.316Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.7.1","x-envoy-origin-path":"/json","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.244.0.9","x-request-id":"49838d97-42d1-4913-9f8e-d389b2ddddce"}
Controller logs
2025-06-06T21:18:22.142Z INFO admin admin/server.go:34 starting admin server {"address": "127.0.0.1:19000", "enablePprof": false}
2025-06-06T21:18:22.143Z INFO metrics metrics/register.go:179 initialized metrics pull endpoint {"address": "0.0.0.0:19001", "endpoint": "/metrics"}
2025-06-06T21:18:22.143Z INFO metrics metrics/register.go:62 starting metrics server {"address": "0.0.0.0:19001"}
2025-06-06T21:18:22.143Z INFO cmd/server.go:67 Start runners
2025-06-06T21:18:22.143Z INFO cmd/server.go:277 Starting runner {"name": "provider"}
2025-06-06T21:18:22.146Z INFO provider.controller-runtime.webhook webhook/server.go:183 Registering webhook {"runner": "provider", "path": "/inject-pod-topology"}
2025-06-06T21:18:22.146Z INFO provider kubernetes/controller.go:141 created gatewayapi controller {"runner": "provider"}
2025-06-06T21:18:22.158Z INFO provider kubernetes/controller.go:1524 ServiceImport CRD not found, skipping ServiceImport watch {"runner": "provider"}
2025-06-06T21:18:22.166Z INFO provider kubernetes/controller.go:1878 Watching gatewayAPI related objects {"runner": "provider"}
2025-06-06T21:18:22.168Z INFO provider runner/runner.go:66 Running provider {"runner": "provider", "type": "Kubernetes"}
2025-06-06T21:18:22.168Z INFO cmd/server.go:277 Starting runner {"name": "gateway-api"}
2025-06-06T21:18:22.168Z INFO gateway-api runner/runner.go:91 started {"runner": "gateway-api"}
2025-06-06T21:18:22.168Z INFO cmd/server.go:277 Starting runner {"name": "xds-translator"}
2025-06-06T21:18:22.168Z INFO xds-translator runner/runner.go:53 started {"runner": "xds-translator"}
2025-06-06T21:18:22.168Z INFO cmd/server.go:277 Starting runner {"name": "infrastructure"}
2025-06-06T21:18:22.168Z INFO cmd/server.go:277 Starting runner {"name": "xds-server"}
2025-06-06T21:18:22.168Z INFO xds-server runner/runner.go:98 loaded TLS certificate and key {"runner": "xds-server"}
2025-06-06T21:18:22.168Z INFO provider manager/server.go:83 starting server {"runner": "provider", "name": "health probe", "addr": "[::]:8081"}
2025-06-06T21:18:22.168Z INFO xds-server runner/runner.go:149 started {"runner": "xds-server"}
2025-06-06T21:18:22.168Z INFO provider.controller-runtime.metrics server/server.go:208 Starting metrics server {"runner": "provider"}
2025-06-06T21:18:22.168Z INFO provider.controller-runtime.metrics server/server.go:247 Serving metrics server {"runner": "provider", "bindAddress": ":8080", "secure": false}
2025-06-06T21:18:22.168Z INFO provider.controller-runtime.webhook webhook/server.go:191 Starting webhook server {"runner": "provider"}
2025-06-06T21:18:22.168Z INFO provider.controller-runtime.certwatcher certwatcher/certwatcher.go:211 Updated current TLS certificate {"runner": "provider"}
2025-06-06T21:18:22.168Z INFO provider.controller-runtime.webhook webhook/server.go:242 Serving webhook server {"runner": "provider", "host": "", "port": 9443}
2025-06-06T21:18:22.169Z INFO provider.controller-runtime.certwatcher certwatcher/certwatcher.go:133 Starting certificate poll+watcher {"runner": "provider", "interval": "10s"}
2025-06-06T21:18:22.174Z INFO wasm-cache wasm/httpserver.go:111 Listening on :18002
2025-06-06T21:18:22.270Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha1.HTTPRouteFilter"}
2025-06-06T21:18:22.271Z INFO provider leaderelection/leaderelection.go:257 attempting to acquire leader lease envoy-gateway-system/5b9825d2.gateway.envoyproxy.io... {"runner": "provider"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "*kubernetes.watchAndReconcileSource"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.GatewayClass"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.ConfigMap"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha2.UDPRoute"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.Deployment"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.DaemonSet"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.Gateway"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha1.ClientTrafficPolicy"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.HTTPRoute"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha2.TCPRoute"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.GRPCRoute"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha1.BackendTrafficPolicy"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha2.TLSRoute"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.Service"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha1.EnvoyProxy"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.EndpointSlice"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha1.EnvoyExtensionPolicy"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.Node"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha1.SecurityPolicy"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1alpha3.BackendTLSPolicy"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1.Secret"}
2025-06-06T21:18:22.271Z INFO provider controller/controller.go:204 Starting EventSource {"runner": "provider", "controller": "gatewayapi-1749244702", "source": "kind source: *v1beta1.ReferenceGrant"}
2025-06-06T21:18:22.371Z INFO provider kubernetes/predicates.go:41 gatewayclass has matching controller name, processing {"runner": "provider", "name": "public"}
2025-06-06T21:18:22.372Z INFO provider controller/controller.go:239 Starting Controller {"runner": "provider", "controller": "gatewayapi-1749244702"}
2025-06-06T21:18:22.372Z INFO provider controller/controller.go:248 Starting workers {"runner": "provider", "controller": "gatewayapi-1749244702", "worker count": 1}
2025-06-06T21:18:22.372Z INFO provider kubernetes/controller.go:190 reconciling gateways {"runner": "provider"}
2025-06-06T21:18:22.372Z INFO provider kubernetes/controller.go:2029 processing EnvoyProxy {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public-gateway-config"}
2025-06-06T21:18:22.372Z INFO provider kubernetes/controller.go:1072 processing Gateway {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public"}
2025-06-06T21:18:22.372Z INFO provider kubernetes/routes.go:234 processing HTTPRoute {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:22.473Z INFO provider kubernetes/controller.go:774 processing OIDC HMAC Secret {"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy-oidc-hmac"}
2025-06-06T21:18:22.573Z INFO provider kubernetes/controller.go:423 processing Backend {"runner": "provider", "kind": "Service", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:22.574Z INFO provider kubernetes/controller.go:437 added Service to resource tree {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:22.675Z INFO provider kubernetes/controller.go:362 reconciled gateways successfully {"runner": "provider"}
2025-06-06T21:18:22.675Z INFO provider kubernetes/controller.go:190 reconciling gateways {"runner": "provider"}
2025-06-06T21:18:22.676Z INFO provider kubernetes/controller.go:2029 processing EnvoyProxy {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public-gateway-config"}
2025-06-06T21:18:22.676Z INFO provider kubernetes/controller.go:1072 processing Gateway {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public"}
2025-06-06T21:18:22.676Z INFO provider kubernetes/routes.go:234 processing HTTPRoute {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:22.676Z INFO provider kubernetes/controller.go:774 processing OIDC HMAC Secret {"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy-oidc-hmac"}
2025-06-06T21:18:22.676Z INFO provider kubernetes/controller.go:423 processing Backend {"runner": "provider", "kind": "Service", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:22.676Z INFO provider kubernetes/controller.go:437 added Service to resource tree {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:22.676Z INFO provider kubernetes/controller.go:362 reconciled gateways successfully {"runner": "provider"}
2025-06-06T21:18:22.676Z INFO gateway-api runner/runner.go:129 received an update {"runner": "gateway-api"}
2025-06-06T21:18:22.680Z INFO xds-translator runner/runner.go:61 received an update {"runner": "xds-translator"}
2025-06-06T21:18:22.684Z INFO xds-server runner/runner.go:195 received an update {"runner": "xds-server"}
2025-06-06T21:18:35.860Z INFO xds-server v3/simple.go:569 open delta watch ID:1 for type.googleapis.com/envoy.config.cluster.v3.Cluster Resources:map] from nodeID: "envoy-envoy-gateway-system-public-5ea56ca9-6b454dc6cb-5tzkw", version ""
2025-06-06T21:18:35.864Z INFO xds-server v3/simple.go:569 open delta watch ID:2 for type.googleapis.com/envoy.config.listener.v3.Listener Resources:map] from nodeID: "envoy-envoy-gateway-system-public-5ea56ca9-6b454dc6cb-5tzkw", version "1"
2025-06-06T21:18:35.868Z INFO xds-server v3/simple.go:569 open delta watch ID:3 for type.googleapis.com/envoy.config.route.v3.RouteConfiguration Resources:map[envoy-gateway-system/public/http:{}] from nodeID: "envoy-envoy-gateway-system-public-5ea56ca9-6b454dc6cb-5tzkw", version "1"
2025-06-06T21:18:36.433Z INFO xds-server v3/simple.go:569 open delta watch ID:4 for type.googleapis.com/envoy.config.cluster.v3.Cluster Resources:map] from nodeID: "envoy-envoy-gateway-system-public-5ea56ca9-6b454dc6cb-d9rtk", version ""
2025-06-06T21:18:36.435Z INFO xds-server v3/simple.go:569 open delta watch ID:5 for type.googleapis.com/envoy.config.listener.v3.Listener Resources:map] from nodeID: "envoy-envoy-gateway-system-public-5ea56ca9-6b454dc6cb-d9rtk", version "1"
2025-06-06T21:18:36.435Z INFO xds-server v3/simple.go:569 open delta watch ID:6 for type.googleapis.com/envoy.config.route.v3.RouteConfiguration Resources:map[envoy-gateway-system/public/http:{}] from nodeID: "envoy-envoy-gateway-system-public-5ea56ca9-6b454dc6cb-d9rtk", version "1"
2025-06-06T21:18:38.724Z INFO provider leaderelection/leaderelection.go:271 successfully acquired lease envoy-gateway-system/5b9825d2.gateway.envoyproxy.io {"runner": "provider"}
2025-06-06T21:18:38.724Z INFO provider kubernetes/status_updater.go:134 started status update handler {"runner": "provider"}
2025-06-06T21:18:38.724Z INFO provider kubernetes/controller.go:190 reconciling gateways {"runner": "provider"}
2025-06-06T21:18:38.725Z INFO infrastructure runner/runner.go:75 started {"runner": "infrastructure"}
2025-06-06T21:18:38.725Z INFO provider kubernetes/controller.go:2029 processing EnvoyProxy {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public-gateway-config"}
2025-06-06T21:18:38.725Z INFO provider kubernetes/controller.go:1072 processing Gateway {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public"}
2025-06-06T21:18:38.725Z INFO provider kubernetes/routes.go:234 processing HTTPRoute {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:38.725Z INFO provider kubernetes/controller.go:774 processing OIDC HMAC Secret {"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy-oidc-hmac"}
2025-06-06T21:18:38.725Z INFO provider kubernetes/controller.go:423 processing Backend {"runner": "provider", "kind": "Service", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:38.725Z INFO provider kubernetes/status_updater.go:145 received a status update {"runner": "provider", "namespace": "", "name": "public"}
2025-06-06T21:18:38.725Z INFO infrastructure runner/runner.go:100 received an update {"runner": "infrastructure"}
2025-06-06T21:18:38.725Z INFO provider.public kubernetes/status_updater.go:109 status unchanged, bypassing update {"runner": "provider"}
2025-06-06T21:18:38.725Z INFO provider kubernetes/controller.go:437 added Service to resource tree {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:38.726Z INFO provider kubernetes/controller.go:362 reconciled gateways successfully {"runner": "provider"}
2025-06-06T21:18:38.726Z INFO provider kubernetes/status_updater.go:145 received a status update {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:38.726Z INFO gateway-api runner/runner.go:129 received an update {"runner": "gateway-api"}
2025-06-06T21:18:38.727Z INFO provider.httpbin.envoy-gateway-system kubernetes/status_updater.go:109 status unchanged, bypassing update {"runner": "provider"}
2025-06-06T21:18:38.727Z INFO provider kubernetes/status_updater.go:145 received a status update {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public"}
2025-06-06T21:18:38.734Z INFO provider kubernetes/status_updater.go:145 received a status update {"runner": "provider", "namespace": "envoy-gateway-system", "name": "httpbin"}
2025-06-06T21:18:38.735Z INFO provider.httpbin.envoy-gateway-system kubernetes/status_updater.go:109 status unchanged, bypassing update {"runner": "provider"}
2025-06-06T21:18:38.735Z INFO provider kubernetes/status_updater.go:145 received a status update {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public"}
2025-06-06T21:18:41.834Z INFO provider kubernetes/status_updater.go:145 received a status update {"runner": "provider", "namespace": "envoy-gateway-system", "name": "public"}