Skip to content

Conversation

@sofiaz11
Copy link
Collaborator

@sofiaz11 sofiaz11 commented Nov 20, 2025

Problem Statement

Skip the parent VT truncate based on the concurrent push strategy

Solution

Add the check on the top level calls of truncateKafkaTopic:
VeniceHelixAdmin class:

  • deleteOneStoreVersion
  • cleanUpViewResource (constructViewResources: view resource is not created in parent fabric)

VeniceParentHelixAdmin class:

  • truncateKafkaTopic (overridden the VeniceHelixAdmin's method, kept the method)
  • truncateTopicBasedOnMaxErroredTopicNumToKeep (will be removed once ramp to version status based tracking)
  • rollForwardToFutureVersion
  • truncateTopicsOptionally
  • killOfflinePush

Code changes

  • Added new code behind a config. If so list the config names and their default values in the PR description.
  • Introduced new log lines.
    • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

  • Code has no race conditions or thread safety issues.
  • Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
  • No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
  • Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
  • Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

  • New unit tests added.
  • New integration tests added.
  • Modified or extended existing tests.
  • Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

  • No. You can skip the rest of this section.
  • Yes. Clearly explain the behavior change and its impact.

@sofiaz11 sofiaz11 marked this pull request as ready for review November 24, 2025 19:46
Copilot AI review requested due to automatic review settings November 24, 2025 19:46
Copilot finished reviewing on behalf of sofiaz11 November 24, 2025 19:51
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces conditional logic to skip parent Venice topic (VT) truncation based on the concurrent push detection strategy. Specifically, when using PARENT_VERSION_STATUS_ONLY strategy (which doesn't require parent VT writes), the parent controller will skip topic truncation operations to avoid unnecessary overhead.

Key Changes:

  • Added isTruncatingTopicNeeded() method to check if topic truncation should occur based on controller type and concurrent push strategy
  • Wrapped topic truncation calls in VeniceHelixAdmin and VeniceParentHelixAdmin with conditional checks
  • Updated test mocks to properly stub the new isTruncatingTopicNeeded() method

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java Added isTruncatingTopicNeeded() method and wrapped version topic truncation in deleteOneStoreVersion() with conditional check
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceParentHelixAdmin.java Added conditional checks before topic truncation in rollForwardToFutureVersion(), truncateTopicsOptionally(), and killOfflinePush() methods
services/venice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceParentHelixAdmin.java Updated test mocks to stub isTruncatingTopicNeeded() method returning true in relevant test cases
Comments suppressed due to low confidence (1)

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java:4172

  • The truncation of the stream reprocessing topic should also be wrapped with the isTruncatingTopicNeeded(clusterName) check for consistency with the version topic truncation above. When the concurrent push detection strategy is PARENT_VERSION_STATUS_ONLY, parent controllers should skip truncating all related topics, including stream reprocessing topics.
          if (deletedVersion.get().getPushType().isStreamReprocessing()) {
            truncateKafkaTopic(Version.composeStreamReprocessingTopic(storeName, versionNumber));
          }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ontroller/VeniceHelixAdmin.java

Co-authored-by: Copilot <[email protected]>
Copilot AI review requested due to automatic review settings November 25, 2025 23:23
…ontroller/VeniceHelixAdmin.java

Co-authored-by: Copilot <[email protected]>
Copilot finished reviewing on behalf of sofiaz11 November 25, 2025 23:28
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java:4172

  • The stream reprocessing topic truncation at line 4171 is not guarded by isTruncatingTopicNeeded(clusterName), while the main version topic truncation at lines 4162-4168 is properly guarded. This creates an inconsistency where the parent controller might skip truncating the main version topic but still truncate the stream reprocessing topic. The guard should be applied consistently to both topic types.

Consider wrapping line 4171 with the same conditional check:

if (isTruncatingTopicNeeded(clusterName) && deletedVersion.get().getPushType().isStreamReprocessing()) {
  truncateKafkaTopic(Version.composeStreamReprocessingTopic(storeName, versionNumber));
}
          if (deletedVersion.get().getPushType().isStreamReprocessing()) {
            truncateKafkaTopic(Version.composeStreamReprocessingTopic(storeName, versionNumber));
          }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

boolean onlyDeferredSwap =
futureVersion.isVersionSwapDeferred() && StringUtils.isEmpty(futureVersion.getTargetSwapRegion());
if (onlyDeferredSwap) {
if (onlyDeferredSwap && !getVeniceHelixAdmin().shouldSkipTruncatingTopic(clusterName)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in parent controller just checking getConcurrentPushDetectionStrategy should be enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants