HDDS-13780. Skeleton of background snapshot defrag service #9133
+502
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit f81e5b7)
(cherry picked from commit e15a62d)
…efrag.limit.per.task`, `ozone.snapshot.defrag.service.interval`. (cherry picked from commit c04d67a)
(cherry picked from commit ac58394)
(cherry picked from commit ab9d67e)
Conflicts: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java
… be handled in upper logic
There was a problem hiding this comment.
Pull Request Overview
This PR implements a skeleton for a background snapshot defragmentation service in Ozone Manager. The service is designed to process snapshots in the active snapshot chain and defragment them by creating compacted RocksDB instances.
- Adds the main SnapshotDefragService class with background task scheduling and snapshot processing logic
- Integrates the service into KeyManager with configuration and lifecycle management
- Updates related classes to support the defragmentation infrastructure
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| SnapshotDefragService.java | New background service implementing snapshot defragmentation with full and incremental processing modes |
| OmSnapshotLocalDataYaml.java | Fixed null handling for previous snapshot ID in YAML data construction |
| KeyManagerImpl.java | Added service startup/shutdown and configuration integration for snapshot defragmentation |
| KeyManager.java | Added interface method for accessing snapshot defrag service |
| OMConfigKeys.java | Added configuration keys for defrag service interval and limits |
| RocksDatabase.java | Made checkpoint creation and get methods public for external access |
| RDBSstFileWriter.java | Made class and constructor public, added delete method for SST file operations |
| ozone-default.xml | Added configuration properties for snapshot defragmentation service |
| OzoneConsts.java | Added constant for defragmented checkpoint directory |
| OzoneConfigKeys.java | Added timeout configuration for snapshot defragmentation service |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Show resolved
Hide resolved
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
Show resolved
Hide resolved
swamirishi
reviewed
Oct 9, 2025
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
TaiJuWu
reviewed
Oct 11, 2025
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBSstFileWriter.java
Show resolved
Hide resolved
swamirishi
requested changes
Oct 13, 2025
There was a problem hiding this comment.
@smengcl thank you for working on the patch have a few review comments
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBSstFileWriter.java
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
Thanks @swamirishi for the review. I have addressed all comments. Pls take another look. |
swamirishi
approved these changes
Oct 14, 2025
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotDefragService.java
Outdated
Show resolved
Hide resolved
swamirishi
reviewed
Oct 14, 2025
| } | ||
|
|
||
| private UncheckedAutoCloseableSupplier<OmSnapshot> getSnapshot(String snapshotTableKey, boolean skipActiveCheck) | ||
| UncheckedAutoCloseableSupplier<OmSnapshot> getSnapshot(String snapshotTableKey, boolean skipActiveCheck) |
There was a problem hiding this comment.
@smengcl No need for this change anymore
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is #9117 but shrunk down to just the skeleton of the service, without actual defrag operations.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13780
How was this patch tested?