Skip to content

[controller] Increase dead store stats prefetch timeout to 5 minutes #2232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[controller] Increase dead store stats prefetch timeout to 5 minutes #2232

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4d579c4
[controller] increase dead store stats prefetch timeout to 5 minutes
Oct 22, 2025
d838fd1
Skip if no longer leader
Oct 22, 2025
d544e57
feat: increase dead store stats prefetch timeout to 5 minutes to acco…
Oct 22, 2025
592ded0
feat: handle thread interruption in dead store stats prefetch for gra…
Oct 22, 2025
ccf1894
Adding interrupt check
Oct 22, 2025
e51b210
refactor: use Thread.sleep with InterruptedException for cleaner inte…
Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions ...e-controller/src/main/java/com/linkedin/venice/controller/DeadStoreStatsPreFetchTask.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,16 @@ public void run() {
logger.info("Started {}", taskId);
isRunning.set(true);
try {
// wait for 60 seconds for controller to become leader as it's required to be leader for dead store stats
long deadline = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(60);
// wait for 300 seconds for controller to become leader as it's required to be leader for dead store stats
long deadline = System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(300);
while (!admin.isLeaderControllerFor(clusterName)) {
if (System.currentTimeMillis() > deadline) {
throw new VeniceException("Timed out waiting for controller to become leader for cluster: " + clusterName);
}
Utils.sleep(10_000); // sleep for 10 seconds
if (!Utils.sleep(10_000)) {
logger.info("Sleep was interrupted, stopping dead store stats prefetch task for cluster: {}", clusterName);
break;
}
}

logger.debug("Initial fetch of dead store stats for cluster: {}", clusterName);
Expand All @@ -58,7 +61,11 @@ public void run() {

while (isRunning.get()) {
try {
Utils.sleep(refreshIntervalMs);
// Check return value of sleep to detect thread interruption for graceful shutdown
if (!Utils.sleep(refreshIntervalMs)) {
Copy link
Contributor

@majisourav99 majisourav99 Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its cleaner to call Thread.sleep and capture the exception without this extra check on utils.sleep

logger.info("Sleep was interrupted, stopping dead store stats prefetch task for cluster: {}", clusterName);
break;
}
long startTime = System.currentTimeMillis();
logger.debug("Fetching dead store stats for cluster: {}", clusterName);
admin.preFetchDeadStoreStats(clusterName, getStoresInCluster());
Expand Down
Loading