Draft: AI Policies

[jasonmadigan]

Re: #118

Not ready for review yet.

[maleck13]

Would be interesting to get some feedback from a SME about what they and where they want things checked. In my naive view I can see the gateway being a good place to check the prompt. But I wonder about the response. Why would you not do that to be done before sending the response over the network?

[david-martin]

It may not be possible to avoid network traffic, like in the case of streaming the response.
You may end up streaming/chunking the response and checking it all, or in parts.

[maleck13]

My instinct here as hinted at above is to start with the Prompt check and seek more opinion and feedback on the suitability of checking this at the gateway? Maybe it is still desirable to have a "last line of defence" type policy in place?

[david-martin]

Depending on how you use it, it could be a last line of defence.

It could also be a targeted response check based on a specific group of users for example, since you have access to the users auth context. That capability may not be easily available in the model serving runtime without custom logic.

[maleck13]

I wonder could these be in the same API such as LLMContentPolicy but split underneath by the kuadrant operator as they are very similar? or do you expect the model being used to change based on the policy type?

spec:
  model: ....
  llmprompt:
    categories:
       ....
  llmresponse:
    categories:
response:
...

[david-martin]

They certainly could be in the same API.
My sole reason at the time for splitting them was usability.
That is, writing/configuring each part separately in a sizable block.

There may be a more concrete case for keeping them split.
Thoughts on if you want to apply a prompt check to a different group of users than a risk check?
Although not shown in this example, it could be useful to have a predicate for what users the policy applies to (based on auth headers etc..)
If combined into 1 policy, it would mean having multiple predicate fields.
Do we have a precedence for this with existing policies?

[maleck13]

yes RLP has that

  limits:
    "alice-limit":
      rates:

      - limit: 5
        window: 10s
      when:
      - predicate: "auth.identity.userid == 'alice'"
    "bob-limit":
      rates:
      - limit: 2
        window: 10s
      when:
      - predicate: "auth.identity.userid == 'bob'"

[maleck13]

so perhaps its a use case like

spec:
  model: ....
  llmprompt:
    - "under18":
       categories:
          ....
       when:
        - predicate: "auth.identity.age < 18"
      - "over18"
       categories:
          ....
       when:
        - predicate: "auth.identity.age >= 18" 
        - predicate: "request.model == 'educational'"
  llmresponse:
   -  "under18":

This is just sudo stuff but might be useful to think about?

[alexsnaps]

nit: request.model seem to be wanting to "enhance" the request struct with arbitrary fields. I'd advise against that. When is that model field present? I don't want users to ask themselves these questions. I understand this is just for illustration purposes, but nonetheless raises an interesting question: where would "additional policies" append data to the "well-known attributes"?

[maleck13]

Yes where we append this data is an important consideration. Perhaps we need a new namespace for AI metadata. ai.model

[maleck13]

I don't know if this is needed in the RLP api? it seems a very special use case. I wonder instead if it could be made some form of internal config not exposed to the user at this point?

[jasonmadigan]

that is an option yes: we either inc the counter in a custom fashion in something in this filter chain, or we extend RLP somehow to support. if there's general utility in RLP, I guess that route may be more preferable

slight worry about having another mechanism other than limitador doing it - would end up needing to re-implment/copy a bunch of existing machinery? unsure

[maleck13]

hmm I might be misunderstanding. I was instead thinking of using the hits_addend and setting it dynamically if it is an AI interaction. We need @eguzki or @alexsnaps here interested to know what there thoughts are on how to send a custom increment.

[alexsnaps]

While this isn't implemented afaik, well-known attributes were meant to support this. So one way would be to have ratelimit.hits_addend (while defaulting to 1) be mutable by "upstream" actions. So that a policy could set it to some arbitrary value before the request to limitador is made.

[jasonmadigan]

that'd do nicely

[alexsnaps]

and, yes! 🙌 , service.ext_proc.v3.ProcessingResponse support dynamic metadata to have data flow between envoy and these processes. If wasm-shim dispatches the call, then no worries in using that for that. If envoy does, we have to check what happens to it and how/if we can read them back properly from wasm tho (should be fine™ - t&c apply)

[maleck13]

is there one of these that is more popular than the other? I am wondering does it make sense to support just one for now and expand beyond that in the future?

[jasonmadigan]

assume this is re: Chat completion API vs Responses API

The chat completion API is more universally supported, the Responses API is newer, but is designed to work with more use-cases (agentic use-cases, as well as support for "reasoning/show my thoughts" response streaming)

[maleck13]

beyond these tokens are there other reasons a user may want to pull something from the body? I think I would start with inferring and looking for these specific values rather than surfacing them into the API at this stage?

[maleck13]

As in I would prefer the user to have to hint at the response types to expect and we then use that to decide what values to pull out rather than open up the entire request body to the user to pull values out of.

[jasonmadigan]

Reason I think we may want to offer some sort of API here is that although lots of runtimes offer openai-style completion APIs, in the ones I've looked at there are some small differences (i.e. where in the JSON response usage metrics are) which could break the policy. We could have some prebuilt "variants" to make the APIs look nice though (perhaps starting with one for llama-stack and one for openai-style) - these variants would come with builtin selectors on what attributes to pluck to get our usage metrics (if that makes sense)

[maleck13]

Yeah like we could for now call out those as the supported options. Rather than jumping straight to giving the power to the user which we might regret and not be able to take back. That said we are talking about Alpha apis so easier to take back then other places

[jasonmadigan]

that makes sense. I suppose internally we'll probably use selectors, and then if we decide later, we can expose those to end users

[maleck13]

This is a really great start to some cool features. I think we still need to nail down how we want to do the request filtering (seems to be leaning towards the WASM shim) and also whether we need to expose certain options to the user or not

[jasonmadigan]

One other potential policy which may emerge here, depending on how this PoC progresses, is a SemanticCachingPolicy for short-circuiting (well, partially, still need to do embedding) expensive LLM calls if we see similar prompts