HTTPRoute backendRefs order affects cluster generation when mixing Service and Backend kinds

[aneesh-wolt]

Description:

When defining multiple backends in backendRefs in an HTTPRoute that mix Service and Backend (CRD) kinds, the order of the backends matters. If a Service type backend is listed first and a Backend CRD is listed second, the backends fail to generate proper Envoy clusters, resulting in "no healthy upstream" errors.

I have a use case for the fixes
#7950
#7826

Environment:

• Envoy Gateway version: latest
• Kubernetes version: v1.32.9-eks-3025e55

Repro steps:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test-mixed-backends
spec:
  parentRefs:
    - name: envoy-gateway
      namespace: envoy-gateway
  hostnames:
    - example.com
  rules:
    - matches:
      - path:
          type: PathPrefix
          value: /
      backendRefs:
        - name: local-backend-svc         # Service type FIRST
          port: 80
          weight: 1
          filters:
            - type: URLRewrite
              urlRewrite:
                hostname: "example.com"
        - group: gateway.envoyproxy.io  # Backend CRD SECOND
          kind: Backend
          name: external-backend
          weight: 1
          filters:
            - type: URLRewrite
              urlRewrite:
                hostname: "external-backend.foo.bar.com"
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: external-backend
spec:
  endpoints:
    - fqdn:
        hostname: external-backend.foo.bar.com
        port: 8443
  tls:
    sni: external-backend.foo.bar.com
    wellKnownCACertificates: System

The order of backendRefs should not affect whether the backends are correctly translated into Envoy clusters. Both configurations should produce identical functional results.

Working config:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test-mixed-backends
spec:
  parentRefs:
    - name: envoy-gateway
      namespace: envoy-gateway
  hostnames:
    - example.com
  rules:
    - matches:
      - path:
          type: PathPrefix
          value: /
      backendRefs:
        - group: gateway.envoyproxy.io  # Backend CRD FIRST
          kind: Backend
          name: external-backend
          weight: 50
        - name: local-backend-svc       # Service type SECOND
          port: 80
          weight: 50

Actual Behavior

• Non-working (Service first): Generates separate clusters per backend (rule/0/backend/0, rule/0/backend/1), but the Service-based cluster lacks proper endpoint configuration, resulting in "no healthy upstream" errors.
• Working (Backend first): Generates a single weighted cluster (rule/0) with both backends as endpoints in the load_assignment.

The issue appears to be in the NeedsClusterPerSetting() function and related logic in the IR translation layer.

[aneesh-wolt]

...

Root Cause Analysis

The issue appears to be in the NeedsClusterPerSetting() function and related logic in the IR translation layer.

Relevant Source Files

1. internal/ir/xds.go - Lines 1667-1693

func (r *RouteDestination) NeedsClusterPerSetting() bool {
    return r.HasMixedEndpoints() ||
        r.HasFiltersInSettings() ||
        (len(r.Settings) > 1 && r.HasPreferLocalZone())
}

func (r *RouteDestination) HasMixedEndpoints() bool {
    destinationAddressTypes := sets.Set[DestinationAddressType]{}
    for _, s := range r.Settings {
        if s.AddressType != nil {
            destinationAddressTypes.Insert(*s.AddressType)
        }
    }
    return destinationAddressTypes.Len() > 1 || destinationAddressTypes.Has(MIXED)
}

2. internal/gatewayapi/route.go - Lines 1797-1812 in processDestination()

The processDestination() function handles Service and Backend kinds differently:

• For Service: Calls processServiceDestinationSetting() which sets AddressType based on EndpointSlice
• For Backend: Calls processBackendDestinationSetting() which sets AddressType based on FQDN/IP

3. internal/xds/translator/translator.go - Lines 607-627

if !httpRoute.NeedsClusterPerSetting() {
    err = processXdsCluster(
        tCtx,
        httpRoute.Destination.Name,
        httpRoute.Destination.Settings,
        // ... creates single cluster
    )
} else {
    // Creates separate cluster per setting
    for _, setting := range httpRoute.Destination.Settings {
        tSettings := []*ir.DestinationSetting{setting}
        err = processXdsCluster(
            tCtx,
            setting.Name,  // Uses setting.Name which includes /backend/N
            tSettings,
            // ...
        )
    }
}

The Bug

When processing mixed Service and Backend kinds:

1. The AddressType is populated differently for each kind
2. When Service is processed first, its AddressType may be nil or set to a value that doesn't trigger HasMixedEndpoints()
3. The HasMixedEndpoints() check doesn't properly detect the mixed endpoint types when the order is Service-first
4. This causes NeedsClusterPerSetting() to return inconsistent values based on order
5. When it returns true (per-backend clusters), the cluster generation for the Service-based backend fails to include proper endpoint configuration

Workaround

Either:

1. Put Backend CRD references before Service references in backendRefs
2. Add filters to all backends (this forces HasFiltersInSettings() to return true, creating separate clusters that work correctly)

Suggested Fix

The HasMixedEndpoints() function should be order-independent and should properly detect when Service and Backend kinds are mixed, regardless of their order in the backendRefs list.

Additional Context

Config Dump Comparison

Non-working config creates separate clusters:

• httproute/development/test-mixed-backends/rule/0/backend/0 (Service - missing proper endpoints)
• httproute/development/test-mixed-backends/rule/0/backend/1 (Backend - has endpoints)

Working config creates a single weighted cluster:

• httproute/development/test-mixed-backends/rule/0 (with both backends as localities in load_assignment)

[arkodg]

cc @zirain @zhaohuabing

[zhaohuabing]

I dug into this a bit further, and it turns out the issue is caused by invalid endpoints generated by the xds translator:

[2026-01-20 03:42:08.884][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment rejected: malformed IP address: httpbin.org. Consider setting resolver_name or setting cluster type to 'STRICT_DNS' or 'LOGICAL_DNS'

- clusterName: httproute/default/mixed-service-and-backend/rule/0/backend/1
  endpoints:
  - lbEndpoints:
    - endpoint:
        address:
          socketAddress:
            address: httpbin.org      # should use 'STRICT_DNS' instead of EDS
            portValue: 80
      loadBalancingWeight: 1
    loadBalancingWeight: 1
    locality:
      region: httproute/default/mixed-service-and-backend/rule/0/backend/1