Skip to content

Conversation

@yibin87
Copy link
Contributor

@yibin87 yibin87 commented Sep 29, 2025

What problem does this PR solve?

Issue Number: close #62920

Problem Summary:
Add client network in/out stats in topsql. And record topN network usage sqls adding to current topN cpu usage sqls. Note one sql will be added at most once.

What changed and how does it work?

  1. Add two new session variable "InPacketBytes", "OutPacketBytes" to record the network traffic from/to clients.
  2. When query finishes, record the network traffic info to topsql stmtstats
  3. Pick topN query with highest network traffic.

Check List

Sysbench Performance Test

The performance impact of this PR is not noticable.
Turn off topsql
img_v3_02ro_ba405d95-0fc4-4a72-a4d5-22025a97061g

Turn on topsql
img_v3_02ro_47c2617c-270e-415a-8df5-2ab18fa94a4g

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Run tpch benchmark, print topsql report data on tidb side, and check the network in/out data using sql digest manually to ensure network data make sense.
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Collects client network data for topsql to pick topN sql with highest network traffic.

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 29, 2025
@tiprow
Copy link

tiprow bot commented Sep 29, 2025

Hi @yibin87. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Sep 29, 2025
@codecov
Copy link

codecov bot commented Sep 29, 2025

Codecov Report

❌ Patch coverage is 87.50000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.4881%. Comparing base (de78786) to head (b3321af).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #63796        +/-   ##
================================================
+ Coverage   72.7426%   73.4881%   +0.7455%     
================================================
  Files          1859       1860         +1     
  Lines        504040     504265       +225     
================================================
+ Hits         366652     370575      +3923     
+ Misses       115130     111634      -3496     
+ Partials      22258      22056       -202     
Flag Coverage Δ
integration 41.8553% <15.0000%> (?)
unit 72.6466% <87.5000%> (+0.3361%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.8700% <ø> (ø)
parser ∅ <ø> (∅)
br 46.4135% <ø> (+0.0859%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@yibin87 yibin87 changed the title topsql: Collect client network info for topsql and pick another topN network sql topsql: Collect client network info for topsql and pick another topN sql with highest network traffic Sep 29, 2025
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 30, 2025
@ti-chi-bot ti-chi-bot bot added sig/planner SIG: Planner size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 11, 2025
@yibin87 yibin87 force-pushed the collect_network_info_for_topsql branch from 64f35fc to 36587b4 Compare October 11, 2025 07:22
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 11, 2025
@yibin87 yibin87 force-pushed the collect_network_info_for_topsql branch from 36587b4 to 28f7f2d Compare October 11, 2025 07:25
@yibin87
Copy link
Contributor Author

yibin87 commented Oct 11, 2025

/cc @nolouch @zimulala @XuHuaiyu

@ti-chi-bot ti-chi-bot bot requested review from XuHuaiyu, nolouch and zimulala October 11, 2025 07:27
@yibin87 yibin87 changed the title topsql: Collect client network info for topsql and pick another topN sql with highest network traffic *: Collect client network info for topsql and pick another topN sql with highest network traffic Oct 13, 2025
@yibin87 yibin87 changed the title *: Collect client network info for topsql and pick another topN sql with highest network traffic *: Collect client network info for topsql to pick topN sql with highest network traffic Oct 13, 2025
@yibin87 yibin87 requested a review from zimulala October 16, 2025 02:48
@yibin87
Copy link
Contributor Author

yibin87 commented Oct 16, 2025

/test check-dev2

@tiprow
Copy link

tiprow bot commented Oct 16, 2025

@yibin87: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test check-dev2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@yibin87 yibin87 requested a review from zimulala October 16, 2025 05:22
@ti-chi-bot
Copy link

ti-chi-bot bot commented Oct 30, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-10-16 06:15:59.234601512 +0000 UTC m=+334065.311854062: ☑️ agreed by zimulala.
  • 2025-10-30 20:02:54.041723265 +0000 UTC m=+1593280.118975825: ☑️ agreed by nolouch.

Signed-off-by: yibin87 <[email protected]>
Signed-off-by: yibin <[email protected]>
Signed-off-by: yibin87 <[email protected]>
Signed-off-by: yibin87 <[email protected]>
Signed-off-by: yibin87 <[email protected]>
Signed-off-by: yibin87 <[email protected]>
Signed-off-by: yibin87 <[email protected]>
Signed-off-by: yibin87 <[email protected]>
Signed-off-by: yibin87 <[email protected]>
@yibin87 yibin87 force-pushed the collect_network_info_for_topsql branch from 416da1a to b3321af Compare November 5, 2025 01:26
Copy link

@yudongusa yudongusa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also open a document PR for this.

@ti-chi-bot
Copy link

ti-chi-bot bot commented Nov 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nolouch, yudongusa, zimulala

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Nov 5, 2025
@yibin87
Copy link
Contributor Author

yibin87 commented Nov 5, 2025

/test mysql-test

@tiprow
Copy link

tiprow bot commented Nov 5, 2025

@yibin87: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@yibin87
Copy link
Contributor Author

yibin87 commented Nov 5, 2025

/test mysql-test

@tiprow
Copy link

tiprow bot commented Nov 5, 2025

@yibin87: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot merged commit e407943 into pingcap:master Nov 5, 2025
27 checks passed
@yibin87
Copy link
Contributor Author

yibin87 commented Feb 2, 2026

/cherry-pick release-8.5

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Feb 2, 2026
@ti-chi-bot
Copy link
Member

@yibin87: new pull request created to branch release-8.5: #65962.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@yibin87
Copy link
Contributor Author

yibin87 commented Feb 2, 2026

/cherrypick release-8.5

@ti-chi-bot
Copy link
Member

@yibin87: new pull request could not be created: failed to create pull request against pingcap/tidb#release-8.5 from head ti-chi-bot:cherry-pick-63796-to-release-8.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-63796-to-release-8.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request","status":"422"}

Details

In response to this:

/cherrypick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Feb 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/planner SIG: Planner size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collect SQL network usage for topsql

5 participants