feat: add support for InferencePool

[Xunzhuo]

Description

This PR addes support for inferencePool support, which allows Envoy AI Gateway to integrate with ANY endpoint picker who is supported the inferencePool.

By integrating with the Endpoint Picker like Gateway API Inference Extenstion or the non-GIE EPP, it can expand Envoy AI Gateway`s abilities to advanced scheduleing algorithm to optimize inference.

Related Issues/PRs (if applicable)

Fixes: #423
Fixes #604

Some follow-up:

1. Docs: Add User-Guide Docs & Update Blog
2. Functionality: Support Host Override LbPolicy and fallback
3. Testing: Support Upstream Conformance Test

[Xunzhuo]

inference pool and endpoint picker:

aigwroute:

httproute:

auto-generated eep (target to the httproute and refer to the endpoint picker backend):

auto-generate backend (target to the endpoint picker service):

cluster is patched by the extension server, called by the envoy gateway, modifies it to the original dst:

[mathetake]

instead of creating eep, how about adding extproc into the specific route which routes to the inference pool lb policy cluster? that way other normal routes won't need to talk to eep unnecessarily. This could be done in extension server i guess?

[yuzisun]

instead of creating eep, how about adding extproc into the specific route which routes to the inference pool lb policy cluster? that way other normal routes won't need to talk to eep unnecessarily. This could be done in extension server i guess?

That’s a very good point. We need to make sure normal routes do not go through epp.

[Xunzhuo]

Yep, that is reasonable :)

[mathetake]

having said that, one concern about the per route is that it might not work well with ClearRouteCache: true which is set by our AI Gateway extproc. The EPP extproc must come after the ai gateway extproc since until then envoy doesn't know the destination. However, per route filter config might not work well with the deferred route calculation. Maybe it's not the case but something i am worried about it now...

[Xunzhuo]

After envoyproxy/gateway#6524 lands, new algorithm would be:

1. find inferencepool relevant listener
2. insert epp extproc config into listener
3. find unrelated routes under relevant listener
4. insert extproc perroute to disable these routes

[mathetake]

looks good and it would be much simpler

[Xunzhuo]

new approach based on envoyproxy/gateway#6524 landed it in 03573fd, and e2e test passed locally