[Fix] update perf log #1135
[Fix] update perf log #1135
Conversation
Summary of ChangesHello @Yunnglin, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request focuses on enhancing the performance and stability of the application's benchmarking and logging infrastructure. It refines asynchronous execution by ensuring that potentially blocking I/O operations, such as database commits and logging, are offloaded to a separate thread pool, thereby preventing event loop starvation. Additionally, it addresses concurrency concerns with SQLite database access and optimizes resource locking for improved efficiency. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces several improvements to the performance benchmark tool, primarily focusing on making blocking operations non-blocking in an asynchronous context. The changes correctly use asyncio.to_thread for database commits and logging, which prevents the event loop from being blocked. Additionally, the critical section protected by a semaphore has been reduced, which should improve concurrency. I have one suggestion to extend the non-blocking pattern to a GPU utility function to further enhance the accuracy and performance of the benchmark.
| benchmark_data = await client.post(request) | ||
| benchmark_data.update_gpu_usage() | ||
| await benchmark_data_queue.put(benchmark_data) | ||
| benchmark_data.update_gpu_usage() |
There was a problem hiding this comment.
The update_gpu_usage method is a synchronous function. Calling a potentially blocking synchronous function directly within an async function will block the event loop. This can negatively impact performance and the accuracy of benchmark measurements, especially under high concurrency.
To avoid blocking the event loop, you should run this synchronous call in a separate thread using asyncio.to_thread, which is a pattern you've already applied effectively for other blocking calls like con.commit().
| benchmark_data.update_gpu_usage() | |
| await asyncio.to_thread(benchmark_data.update_gpu_usage) |
There was a problem hiding this comment.
Pull request overview
This PR updates performance benchmarking code to improve async handling of blocking operations. The changes move blocking I/O operations (database commits and visualizer logging) to a thread pool executor to prevent blocking the async event loop, and adjusts the semaphore scope in the request handler to reduce the critical section.
Key changes:
- Wrapped blocking database and logging operations with
asyncio.to_thread()to prevent event loop blocking - Reduced semaphore critical section in
send_request()to only protect the HTTP request itself - Added localization improvement to README documentation link
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| evalscope/perf/benchmark.py | Improved async handling by wrapping blocking I/O operations with asyncio.to_thread(), added check_same_thread=False to SQLite connection to support thread pool execution, and reduced semaphore critical section in request handler |
| README.md | Localized Chinese documentation link text from "Chinese Documentation" to "中文文档" |
| # Stream inserts to DB to avoid accumulating all results in memory | ||
| commit_every = args.db_commit_interval | ||
| processed_since_commit = 0 | ||
|
|
There was a problem hiding this comment.
Consider adding a comment to explain why check_same_thread=False is safe here. SQLite connections with this parameter can be risky in multi-threaded environments, but in this case it's safe because there's only one consumer coroutine that processes the queue sequentially. A comment would help future maintainers understand this design decision.
| # NOTE: check_same_thread=False is safe here because this coroutine is the single | |
| # consumer of benchmark_data_queue and performs all DB operations sequentially. | |
| # The connection is not shared across multiple coroutines, and the only work | |
| # offloaded via asyncio.to_thread is con.commit, which still uses this same | |
| # connection in a controlled manner. |
No description provided.