[Bugfix] Forward backend Content-Type in StreamingResponse

[shernshiou]

Problem

When using the vLLM router with vLLM-omni + Qwen3-TTS, audio streams returned via /v1/audio/speech get interrupted/truncated. The same TTS request works perfectly when sent directly to the vLLM backend, but fails when routed through the production-stack router.

In src/vllm_router/services/request_service/request.py, the StreamingResponse hardcodes media_type="text/event-stream". This forces SSE framing on binary audio responses, corrupting the audio data and causing clients to misinterpret the stream.

Additionally, process_request attempts to JSON-decode every response chunk for token tracking, which raises UnicodeDecodeError on binary (non-UTF-8) audio payloads and crashes the request pipeline.

Changes

1. Dynamic Content-Type forwarding
   StreamingResponse now uses the backend's Content-Type header instead of always assuming text/event-stream. Falls back to text/event-stream when the header is absent.

2. Graceful handling of binary responses
   Token-tracking now catches UnicodeDecodeError and ValueError alongside json.JSONDecodeError, so binary audio chunks no longer crash the pipeline.

---

• Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

Detailed Checklist (Click to Expand)

Thank you for your contribution to production-stack! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Feat] for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).
• [Router] for changes to the vllm_router (e.g., routing algorithm, router observability, etc.).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• Pass all linter checks. Please use pre-commit to format your code. See README.md for installation.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Please include sufficient tests to ensure the change is stay correct and robust. This includes both unit tests and integration tests.

DCO and Signed-off-by

When contributing changes to this project, you must agree to the DCO. Commits must include a Signed-off-by: header which certifies agreement with the terms of the DCO.

Using -s with git commit will automatically add this header.

What to Expect for the Reviews

We aim to address all PRs in a timely manner. If no one reviews your PR within 5 days, please @-mention one of YuhanLiu11
, Shaoting-Feng or ApostaC.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where the vLLM router was incorrectly handling binary audio streams, leading to truncation and service crashes. The changes ensure that the router accurately forwards the Content-Type header from the backend and robustly processes non-textual data, thereby resolving audio streaming interruptions and improving the stability of the request pipeline for diverse content types.

Highlights

• Dynamic Content-Type Forwarding: Implemented dynamic Content-Type forwarding for StreamingResponse to correctly reflect the backend's media type, falling back to text/event-stream if not specified.
• Graceful Handling of Binary Responses: Enhanced token-tracking in process_request to gracefully handle binary (non-UTF-8) response chunks by catching UnicodeDecodeError and ValueError, preventing pipeline crashes.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• src/tests/test_audio_speech_routing.py
  • Added new test cases to validate dynamic Content-Type forwarding for audio and ensure binary responses do not cause errors.
• src/vllm_router/services/request_service/request.py
  • Expanded error handling for JSON decoding in process_request to include UnicodeDecodeError and ValueError.
  • Modified StreamingResponse creation to dynamically set media_type based on the backend's Content-Type header.

Activity

• No human activity has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively addresses two issues related to handling non-text responses, specifically for audio streaming. The changes are well-implemented and logical. First, dynamically forwarding the Content-Type from the backend instead of hardcoding it ensures that clients receive the correct media type, which is crucial for binary data like audio. Second, gracefully handling decoding errors during token tracking prevents crashes when processing binary response chunks. The addition of a comprehensive test suite in src/tests/test_audio_speech_routing.py is excellent, as it covers the new functionality, regressions, and edge cases, ensuring the fixes are robust. The changes are clean, targeted, and improve the router's reliability.