HDDS-13780. Skeleton of background snapshot defrag service#9133

Merged

smengcl merged 34 commits intoapache:masterfrom

smengcl:HDDS-13780-skeleton-defrag-service

Oct 14, 2025

Contributor

smengcl commented Oct 9, 2025

What changes were proposed in this pull request?

This is #9117 but shrunk down to just the skeleton of the service, without actual defrag operations.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13780

How was this patch tested?

Test case was running fine before the last commit that removes core defrag logic.

smengcl added 19 commits

October 7, 2025 13:35


          HDDS-13009. Background snapshot defrag service

8423ac7


          Make RDBSstFileWriter public to be used in defrag service.

838cd43

(cherry picked from commit f81e5b7)


          Implement delete(key) in RDBSstFileWriter.

61b9ffc

(cherry picked from commit e15a62d)


          Add config ozone.snapshot.defrag.service.timeout, `ozone.snapshot.d…

73625a8

…efrag.limit.per.task`, `ozone.snapshot.defrag.service.interval`.

(cherry picked from commit c04d67a)


          Add comments/TODOs for defrag service interval/timeout

f442502


          Implement SnapshotDefragService

8e37e5d

(cherry picked from commit ac58394)


          Make it compile

08cde3d


          Checkstyle

2cd2cbf


          Add a test case for debugging

6a8adb5

(cherry picked from commit ab9d67e)


          Handle yaml previousSnapshotId null

0a131dd


          Use snapshot chain iterator in findFirstSnapshotNeedingDefrag

668799d


          Merge remote-tracking branch 'asf' into HDDS-13009-defrag-service

93db705

Conflicts:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java


          Handle yaml snapId null so that it won't cause NPE. Null check should…

135100e

… be handled in upper logic


          Remove findPreviousDefraggedSnapshot

f5fd06a


          Remove TODO that is not applicable

e970a3d


          Add new configs to ozone-default.xml

c3de774


          Clean up test case

92256eb


          Logging and TODO cleanups.

9d57b89


          Remove POC defrag logic and test case, keeping the skeleton

4a3e01a

smengcl requested review from Copilot and swamirishi

October 9, 2025 22:05

smengcl added the snapshot label

Copilot AI reviewed

View reviewed changes

Contributor

Copilot AI left a comment

Pull Request Overview

This PR implements a skeleton for a background snapshot defragmentation service in Ozone Manager. The service is designed to process snapshots in the active snapshot chain and defragment them by creating compacted RocksDB instances.

Adds the main SnapshotDefragService class with background task scheduling and snapshot processing logic
Integrates the service into KeyManager with configuration and lifecycle management
Updates related classes to support the defragmentation infrastructure

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
SnapshotDefragService.java	New background service implementing snapshot defragmentation with full and incremental processing modes
OmSnapshotLocalDataYaml.java	Fixed null handling for previous snapshot ID in YAML data construction
KeyManagerImpl.java	Added service startup/shutdown and configuration integration for snapshot defragmentation
KeyManager.java	Added interface method for accessing snapshot defrag service
OMConfigKeys.java	Added configuration keys for defrag service interval and limits
RocksDatabase.java	Made checkpoint creation and get methods public for external access
RDBSstFileWriter.java	Made class and constructor public, added delete method for SST file operations
ozone-default.xml	Added configuration properties for snapshot defragmentation service
OzoneConsts.java	Added constant for defragmented checkpoint directory
OzoneConfigKeys.java	Added timeout configuration for snapshot defragmentation service

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Show resolved Hide resolved

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java Show resolved Hide resolved


          Address applicable comments from copilot

a17a5b6

swamirishi reviewed

View reviewed changes

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

smengcl added 5 commits

October 9, 2025 15:47


          Revert the change that makes RDBSstFileWriter and RocksDatabase public.

2d29d7f


          Empty performFullDefragmentation and performIncrementalDefragmentatio…

c11eda5

…n due to findbugs.


          Remove unused fields

d6a07c4


          Override

66d6378


          Checkstyle

5f1c393


          Config

9d5166a

smengcl requested a review from swamirishi

October 10, 2025 22:44

TaiJuWu reviewed

View reviewed changes

hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBSstFileWriter.java Show resolved Hide resolved

swamirishi requested changes

View reviewed changes

Contributor

swamirishi left a comment

@smengcl thank you for working on the patch have a few review comments

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBSstFileWriter.java Show resolved Hide resolved

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Show resolved Hide resolved

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved

smengcl added 4 commits

October 13, 2025 15:46


          Revert the revert that makes RDBSstFileWriter public.

1c71318


          Refactor. context: apache#9133 (comment)

d6d7c6a


          getSnapshot(snapshotTableKey, false); remove updateSnapshotMetadata

808f0bb


          Use MultiSnapshotLocks.

02f18ce

Contributor Author

smengcl commented Oct 13, 2025

@smengcl thank you for working on the patch have a few review comments

Thanks @swamirishi for the review. I have addressed all comments. Pls take another look.

swamirishi approved these changes

View reviewed changes

Contributor

swamirishi left a comment

LGTM @smengcl thank you for addressing the review comments

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java Outdated Show resolved Hide resolved


          Use getSnapshot(UUID); pmd

2a51f60

swamirishi reviewed

View reviewed changes

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java Outdated Show resolved Hide resolved

smengcl added 3 commits

October 14, 2025 11:18


          Revert package-private

eee47f3


          Merge remote-tracking branch 'asf' into HDDS-13780-skeleton-defrag-se…

48c47e4

…rvice

Conflicts:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java


          Use new API getOmSnapshotLocalData(SnapshotInfo) that comes with th…

f8e3077

…e refactoring

smengcl merged commit 96ce0b6 into apache:master

43 checks passed

smengcl deleted the HDDS-13780-skeleton-defrag-service branch

October 14, 2025 20:06

Contributor Author

smengcl commented Oct 14, 2025

Thanks @swamirishi , @TaiJuWu for reviewing this.

This was referenced Oct 31, 2025

HDDS-13009. Background snapshot defrag service #9227

Closed

HDDS-13009. Background snapshot defrag service (Draft v1) #9117

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels