[controller] Account for max read capacity in quota change by misyel · Pull Request #2235 · linkedin/venice

misyel · 2025-10-23T18:45:22Z

Problem Statement

When a new quota request comes in to the controller, we only validate the value against the default router capacity before approving or rejecting the request. There is a separate max read capacity that is used for router throttling and the correct calculation would be to approve the request if it is within the default router capacity or the max read capacity.

Solution

Approve the quota change if it is within the default router capacity or max read capacity

Code changes

Added new code behind a config. If so list the config names and their default values in the PR description.
Introduced new log lines.
- Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

Code has no race conditions or thread safety issues.
Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

New unit tests added.
New integration tests added.
Modified or extended existing tests.
Verified backward compatibility (if applicable).

New unit test

Does this PR introduce any user-facing or breaking changes?

No. You can skip the rest of this section.
Yes. Clearly explain the behavior change and its impact.

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java

m-nagarajan · 2025-10-28T16:13:51Z

...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java

  private final boolean disableParentRequestTopicForStreamPushes;

  private final int defaultReadQuotaPerRouter;
+  private final long maxReadCapacityCu;


Can you add a comment on what these 2 variables mean and also revisit the name to make it more clear from the name? Lets also add router in the name if it's only for router quota

Good point - added a comment to describe the usage of the two variables and renamed the new config to include router

...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java

m-nagarajan · 2025-10-28T16:19:32Z

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java

+        long maxReadCapacityCu = clusterConfig.getMaxReadCapacityCu();
+        long maxPerRouterCapacity = Math.max(defaultReadQuotaPerRouter, maxReadCapacityCu);
+        long totalClusterCapacity = maxPerRouterCapacity * routerCount;
+        if (Math.max(totalClusterCapacity, maxPerRouterCapacity) < readQuotaInCU.get()) {


totalClusterCapacity will always be >= maxPerRouterCapacity

This isn't true for the parent because the router count is 0 and totalClusterCapacity will be 0. We need to take the max of totalClusterCapacity and maxPerRouterCapacity to correctly account for this case

can you add some comments, if not it looks like a bug (thats what we though when we encountered this if condition before your change 😆 )

Done - added a comment to explain why we need to take the max of total cluster capacity and per router capacity

m-nagarajan · 2025-10-28T19:04:07Z

...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java

        props.getBoolean(CONTROLLER_DISABLE_PARENT_REQUEST_TOPIC_FOR_STREAM_PUSHES, false);
    this.defaultReadQuotaPerRouter =
        props.getInt(CONTROLLER_DEFAULT_READ_QUOTA_PER_ROUTER, DEFAULT_PER_ROUTER_READ_QUOTA);
+    this.maxRouterReadCapacityCu = props.getLong(MAX_READ_CAPACITY, MAX_ROUTER_READ_CAPACITY_CU);


This config needs to be done on both the router and controller from now on?

Also, from router code, I see MAX_READ_CAPACITY with default of 100k and ROUTER_MAX_READ_CAPACITY with default of 6000. how are those different? can we also use the same static variable in router code as well to be consistent?

Yes, it needs to be on both the controller and router from now on so that they share the same value.

ROUTER_MAX_READ_CAPACITY is used as an early throttler before any requests are processed and it will reject the request if the current number of requests for all stores is larger than the configured limit. I believe it's to prevent the router from being overwhelmed from too many requests at once. MAX_READ_CAPACITY is used to distribute the router quota fairly per store and it will decrease each store's quota by a factor if the total store quota is larger than the MAX_READ_CAPACITY value

m-nagarajan · 2025-10-28T19:06:31Z

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java

+        long maxReadCapacityCu = clusterConfig.getMaxReadCapacityCu();
+        long maxPerRouterCapacity = Math.max(defaultReadQuotaPerRouter, maxReadCapacityCu);
+        long totalClusterCapacity = maxPerRouterCapacity * routerCount;
+        if (Math.max(totalClusterCapacity, maxPerRouterCapacity) < readQuotaInCU.get()) {


can you add some comments, if not it looks like a bug (thats what we though when we encountered this if condition before your change 😆 )

github-actions · 2025-11-29T02:10:37Z

Hi there. This pull request has been inactive for 30 days. To keep our review queue healthy, we plan to close it in 7 days unless there is new activity. If you are still working on this, please push a commit, leave a comment, or convert it to draft to signal intent. Thank you for your time and contributions.

github-actions · 2025-12-06T02:10:42Z

Closing this pull request due to 37 days of inactivity. This is not a judgment on the value of the work. If you would like to continue, please reopen or open a new PR and we will be happy to take another look. Thank you again for contributing.

Copilot

Pull request overview

This pull request fixes quota validation logic in the controller to properly account for the maximum read capacity when approving or rejecting quota change requests. Previously, the controller only validated against a default router capacity, but the correct logic should validate against the maximum router read capacity, which is the same value used by routers for actual throttling.

Changes:

Updated quota validation in VeniceHelixAdmin to use max router read capacity instead of default quota per router
Consolidated separate controller and router capacity constants into a single shared constant (MAX_ROUTER_READ_CAPACITY_CU)
Updated router config default from hardcoded 100,000 to 20,000,000 to align with deployed configurations

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
VeniceHelixAdmin.java	Updated quota validation logic to use maxRouterReadCapacityCu and properly handle parent controller edge case (0 live routers)
VeniceControllerClusterConfig.java	Replaced defaultReadQuotaPerRouter field with maxRouterReadCapacityCu, reading from MAX_READ_CAPACITY config key
VeniceRouterConfig.java	Changed default value from hardcoded 100,000 to MAX_ROUTER_READ_CAPACITY_CU constant (20,000,000)
VeniceConstants.java	Renamed DEFAULT_PER_ROUTER_READ_QUOTA to MAX_ROUTER_READ_CAPACITY_CU, updated documentation
TestVeniceHelixAdmin.java	Added new test for quota validation with multiple test cases
Test utility files	Updated references from old constant to new constant

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java

...ces/venice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceHelixAdmin.java

...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java

services/venice-router/src/main/java/com/linkedin/venice/router/VeniceRouterConfig.java

majisourav99 · 2026-02-20T04:13:28Z

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java

+          ZkRoutersClusterManager routersClusterManager = resources.getRoutersClusterManager();
+          int routerCount = routersClusterManager.getLiveRoutersCount();
+          VeniceControllerClusterConfig clusterConfig = resources.getConfig();
+          long totalClusterReadCapacity = clusterConfig.getMaxRouterReadCapacityCu() * routerCount;


Do we want to include server here? Do we have server read capacity estimation ?

We can make the server read capacity estimation as a separate change because it will require ramping and rollout efforts

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-20T22:49:56Z

...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java

+   * Configs to specify the default and max per router quota. This is used in {@link VeniceHelixAdmin} to determine whether
+   * a quota change should be approved or denied and for throttling reads. This value represents the maximum capacity a single
+   * router can handle


The comment mentions "default and max per router quota" but after this PR there is only one config (maxRouterReadCapacityCu). The comment should be updated to say "max per router quota" or "per router read capacity" to accurately reflect that this is a single config value representing the maximum capacity.

Suggested change

* Configs to specify the default and max per router quota. This is used in {@link VeniceHelixAdmin} to determine whether

* a quota change should be approved or denied and for throttling reads. This value represents the maximum capacity a single

* router can handle

* Config to specify the max per router read capacity/quota. This is used in {@link VeniceHelixAdmin} to determine whether

* a quota change should be approved or denied and for throttling reads. This value represents the maximum capacity a single

* router can handle.

majisourav99

LGTM

misyel added 2 commits October 23, 2025 11:37

check for max router capacity for quota

2fa8503

remove wildcare import

c01300a

majisourav99 reviewed Oct 23, 2025

View reviewed changes

...s/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java Outdated Show resolved Hide resolved

misyel added 2 commits October 23, 2025 14:25

move check back inside child

df9ee0a

Merge branch 'main' into quota

54d6f81

m-nagarajan reviewed Oct 28, 2025

View reviewed changes

misyel added 2 commits October 28, 2025 10:25

use static var, add comments

af2a50a

fix test

4d4c28e

m-nagarajan reviewed Oct 28, 2025

View reviewed changes

misyel added 2 commits October 28, 2025 18:56

only use max read capacity config

a73e5e1

set default router quota to 20_000_000

6e24e2f

misyel force-pushed the quota branch from 3bd5fa0 to 6e24e2f Compare October 29, 2025 18:44

fix import

7f508fe

github-actions bot added the stale label Nov 29, 2025

github-actions bot added the closed-by-stale label Dec 6, 2025

github-actions bot closed this Dec 6, 2025

misyel reopened this Feb 20, 2026

Merge branch 'main' into quota

835ac6c

Copilot AI review requested due to automatic review settings February 20, 2026 00:17

Copilot started reviewing on behalf of misyel February 20, 2026 00:17 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

fix build

447d220

github-actions bot removed closed-by-stale stale labels Feb 20, 2026

majisourav99 reviewed Feb 20, 2026

View reviewed changes

fix test

ce65e5e

Copilot AI review requested due to automatic review settings February 20, 2026 22:44

Copilot started reviewing on behalf of misyel February 20, 2026 22:45 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

majisourav99 approved these changes Feb 24, 2026

View reviewed changes

Conversation

misyel commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Statement

Solution

Code changes

Concurrency-Specific Checks

How was this PR tested?

Does this PR introduce any user-facing or breaking changes?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 29, 2025

Uh oh!

github-actions bot commented Dec 6, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

majisourav99 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

misyel commented Oct 23, 2025 •

edited

Loading