Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFE] Event stream format support on /streaming_query API #299

Open
TamiTakamiya opened this issue Jan 21, 2025 · 0 comments
Open

[RFE] Event stream format support on /streaming_query API #299

TamiTakamiya opened this issue Jan 21, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@TamiTakamiya
Copy link
Contributor

Is your feature request related to a problem? Please describe.
I started working on supporting streaming on our UI with the new road-core/service code. I found that when media_type is set to application/json, the /streaming_query API returns JSON data like:

{"event": "token", "data": {"id": 13, "token": " message"}}{"event": "token", "data": {"id": 14, "token": ","}}{"event": "token", "data": {"id": 15, "token": " follow"}}

Can we have an option to receive data in the Event stream format, i.e. adding data: at the beginning and two new lines at the end of each chunk:

data: {"event": "token", "data": {"id": 13, "token": " message"}}

data: {"event": "token", "data": {"id": 14, "token": ","}}

data: {"event": "token", "data": {"id": 15, "token": " follow"}}

I would like to use the fetch-event-source library on our React UI and it is enabled with this change. OpenAI API streaming support is also using the Event stream format (I tested my local ollama server) and I think it is reasonable to support that format on the /streaming_query API. Thanks.

Describe the solution you'd like
Add an option to /streaming_query API to enable the Event stream format.

Describe alternatives you've considered
We (Ansible Lightspeed) are using road-core/service as a backend service. We can convert the streaming format on our service that directly communicates with UI, but we think it is desirable that this is implemented on road-core/service.

Additional context
Chat completions API streaming sample (tested with local ollama server):

$ curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama3.2",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "What is Ansible?"
            }
        ],
        "stream": true
    }'
data: {"id":"chatcmpl-493","object":"chat.completion.chunk","created":1737430585,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":"Ans"},"finish_reason":null}]}

data: {"id":"chatcmpl-493","object":"chat.completion.chunk","created":1737430585,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":"ible"},"finish_reason":null}]}

data: {"id":"chatcmpl-493","object":"chat.completion.chunk","created":1737430585,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":" is"},"finish_reason":null}]}

data: {"id":"chatcmpl-493","object":"chat.completion.chunk","created":1737430585,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":" an"},"finish_reason":null}]}

data: {"id":"chatcmpl-493","object":"chat.completion.chunk","created":1737430585,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":" open"},"finish_reason":null}]}

data: {"id":"chatcmpl-493","object":"chat.completion.chunk","created":1737430585,"model":"llama3.2","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":"-source"},"finish_reason":null}]}
@TamiTakamiya TamiTakamiya added the enhancement New feature or request label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant