Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature COMMANDLOG to record slow and heavy traffic #1294

Open
wants to merge 10 commits into
base: unstable
Choose a base branch
from

Conversation

soloestoy
Copy link
Member

As discussed in PR #336.

We have different types of resources like CPU, memory, network, etc. The slowlog can only record commands eat lots of CPU during the processing phase (doesn't include read/write network time), but can not record commands eat too many memory and network. For example:

  1. run "SET key value(10 megabytes)" command would not be recored in slowlog, since when processing it the SET command only insert the value's pointer into db dict. But that command eats huge memory in query buffer and bandwidth from network. In this case, just 1000 tps can cause 10GB/s network flow.
  2. run "GET key" command and the key's value length is 10 megabytes. The get command can eat huge memory in output buffer and bandwidth to network.

This PR introduces a new command COMMANDLOG, to log commands that consume significant network bandwidth, including both input and output. Users can retrieve the results using COMMANDLOG get <count> heavytraffic-input and COMMANDLOG get <count> heavytraffic-output.

And the slowlog is also incorporated into the commandlog.

Copy link

codecov bot commented Nov 12, 2024

Codecov Report

Attention: Patch coverage is 92.12598% with 10 lines in your changes missing coverage. Please review.

Project coverage is 70.70%. Comparing base (2df56d8) to head (28bd4af).
Report is 1 commits behind head on unstable.

Files with missing lines Patch % Lines
src/commandlog.c 92.79% 8 Missing ⚠️
src/module.c 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #1294      +/-   ##
============================================
+ Coverage     70.69%   70.70%   +0.01%     
============================================
  Files           114      114              
  Lines         63161    63193      +32     
============================================
+ Hits          44650    44680      +30     
- Misses        18511    18513       +2     
Files with missing lines Coverage Δ
src/blocked.c 91.90% <100.00%> (ø)
src/commands.def 100.00% <ø> (ø)
src/config.c 78.83% <ø> (ø)
src/latency.c 80.92% <100.00%> (ø)
src/server.c 87.70% <100.00%> (+<0.01%) ⬆️
src/server.h 100.00% <ø> (ø)
src/module.c 9.66% <0.00%> (-0.01%) ⬇️
src/commandlog.c 92.79% <92.79%> (ø)

... and 10 files with indirect coverage changes

@hwware
Copy link
Member

hwware commented Nov 15, 2024

Just for Note: Let's first refresh the memory in pr #336, the last comment conclusion is:

After a core team meeting, we decided adding a new command COMMANDLOG with subcommands HEAVYTRAFFIC and SLOW, and then slowlog.c can be renamed to a common commandlog.c. #336 (comment)

@hwware
Copy link
Member

hwware commented Nov 15, 2024

From my understanding, for the command: commandlog get should be:
COMMANDLOG get count slow | heavytraffic-input | heavytraffic-output.

commandlog len should be:
COMMANDLOG len slow | heavytraffic-input | heavytraffic-output.

commandlog reset should be:
COMMANDLOG reset slow | heavytraffic-input | heavytraffic-output.

Can you describe the terms heavytraffic-input and heavytraffic-output in the json file (argument part) because I can only know
them from the source codes and valkey.conf so far?

And I think the existing slowlog commands should be deprecated? If yes, I think you should update the related json files as well.

slowlog-max-len 128

# The second type is HEAVYTRAFFIC-INPUT, which is used to log commands with large inputs.
# These commands can consume network bandwidth and the client's query buffer. The input
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# These commands can consume network bandwidth and the client's query buffer. The input
# which can consume network bandwidth and the client's query buffer. The input

heavytraffic-input-max-len 128

# The third type is HEAVYTRAFFIC-OUTPUT, which is used to log commands with large
# execution results. This type consumes network bandwidth and the client's output buffer.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# execution results. This type consumes network bandwidth and the client's output buffer.
# execution results. This kind of commands consume network bandwidth and the client's output buffer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants