Skip to content

Add option to enable dynamic routing in Envoy Proxy #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

kondratyevd
Copy link
Collaborator

One can now set the following parameters in values.yaml:

envoy:
  dynamic_routing:
    enabled: true
  lua_filter:
    enabled: true
    lua_config: "cfg/envoy-filter-dynamic.lua"

In envoy-filter-dynamic.lua, we add logic to inject an HTTP header the value of which contains the address to which the request will be routed by Envoy.

  • For inference requests, we extract model name from request body and redirect the request to model-specific load balancer.
  • For model repository index requests, we will redirect the request to a repository index aggregator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants