-
Notifications
You must be signed in to change notification settings - Fork 943
Data Retention Regulator Change Log
- Bump
Aeron
to 1.48.4. - Bump
Agrona
to 2.2.4. - Bump
Shadow
to 8.3.7. - Bump
JUnit
to 5.13.2. - Bump
Checkstyle
to 10.26.0.
- Bump
Aeron
to 1.48.2. - Bump
JUnit
to 5.13.1.
-
Breaking (behaviour): Changed
RetainEnoughDataForRecoveryPolicy
andMatchingLogRecordingPolicy
to fall back to Aeron's system properties forclusterId
,clusterDirName
, andstandbyDirName
if not set programmatically or via explicit policy properties. -
Breaking (behaviour): Changed the
DataRetentionRegulator#main
entrypoint to make property value precedence more sane. Now, system properties take precedence over property values defined in properties files, and property values defined in "later" files (w.r.t. the command line arguments) take precedence over property values defined in "earlier files. -
Breaking (API): Changed the signature of
DataRetentionRegulator#execute
to return aboolean
indicating whether the data retention regulation was successfully completed or not. It returnsfalse
, e.g., if backing up a segment file was not complete or if an archive operation failed. This allows the caller to retry the operation. -
Breaking (API, behaviour):
DataRetentionRegulator.Context
has had some significant changes:-
#archiveContext
now defaults tonull
. It is initialised to use the local control channels of the archive using the specifiedarchiveDirName
(which are read from its mark file) in theDataRetentionRegulator.Context#conclude
method. -
#aeronDirectoryName
has been removed, as it is now discovered from the archive's mark file. - it no longer extends
CommonContext
.
-
-
Added support for automatic retries in the
DataRetentionRegulator#main
entrypoint (until successful execution) when theaeron.data.retention.execute.timeout
system property is set to a positive value. The default is0
, which will behave as before (i.e., no retries). When set to a positive value, the regulator will retry the operation everyaeron.data.retention.execute.retry.period
(which defaults to 10 seconds) until the operation is successful or the timeout is reached. Both system properties accept time unit suffixes (ns
,us
,ms
, ands
). -
Fixed a bug where
MatchingLogRecordingPolicy
did propagate an evaluation context to the inner policy; therefore, the inner policy might fail due to missing dependencies, e.g., theepochClock
inDeleteSegmentsAfterPeriodPolicy
. -
Added a
retentionPeriod
overload to bothDeleteSegmentsAfterPeriodPolicy.Context
andDeleteRecordingsAfterPeriodPolicy.Context
that accepts amount and unit simultaneously, to avoid the need to call two methods to set the retention period. -
Improved javadoc for
DeleteSegmentsAfterPeriodPolicy.Context
-
Made the retention behaviour of
DeleteSegmentsAfterPeriodPolicy
configurable via a mode. Opting intoretentionMode(RETENTION_UNTIL_DELETION)
makes it possible to compose the policy with others usingRetainMaxOfPolicy
in a way that will keep at least the retention period of data. Previously, and when using modeNO_RETENTION
, the policy would only "cover" recordings when they had segments due for deletion, i.e., segments last modified before the retention period. When using the newRETENTION_UNTIL_DELETION
mode, the policy will "cover" all recordings, i.e., the policy will ensure data is kept for the retention period by proposing a recording start position based on the earliest segment that was modified within the retention period. -
Support providing a backing up strategy via the
aeron.data.retention.backing.up.strategy
system property, where the value should be the name of a class with a no-arg constructor that implementsSegmentFileBackingUpStrategy
. -
Upgrade Aeron to 1.47.5.
-
Upgrade to Aeron Standby 1.47.5.
-
Upgrade to Agrona 2.1.0
-
Breaking:
groupId
changed toio.aeron.premium.drr
:<groupId>io.aeron.premium.drr</groupId> <artifactId>aeron-data-retention-regulator</artifactId>
- Upgrade Aeron to 1.47.4.
- Upgrade to Aeron Standby 1.47.4.
- The
aeron-data-retention-regulator-x.y.z.jar
artifact is now a regular thin jar, the fat jar now lives in a new artifactaeron-data-retention-regulator-x.y.z-all.jar
. - Upgrade Agrona to 2.0.1 - requires additional command line argument.
- Upgrade Aeron to 1.47.2.
- The jars now contain the README and the CHANGELOG:
aeron-data-retention-regulator/README.md aeron-data-retention-regulator/CHANGELOG.md
- Fix a bug where
RecordingMetadata
would be incorrectly populated (introduced in 0.7.0):-
maxRecordedPosition
would be correct only for stopped recordings, -1 for active ones -
startTimestamp
would be themaxRecordedPosition
-
stopTimestamp
would be thestartTimestamp
. Which would have the following effects:-
RetainLengthPolicy
would always retain all active recordings -
DeleteSegmentsAfterPeriodPolicy
would not delete any segments for active recordings -
DeleteRecordingsAfterPeriodPolicy
would delete after period calculated from recording start instead of stop, unless recording is active, then it would fail to delete
-
-
- Upgrade Aeron to 1.46.7.
- Upgrade Agrona to 1.23.1.
- Java 17 or newer is now required.
- JavaDoc jars are now published.
- Upgrade to Aeron 1.46.0
- Upgrade to Agrona 1.23.0
- Upgrade to Aeron 1.45.0
- Upgrade to Agrona 1.22.0
- Make testing policy trees easier in downstream projects without extensive mocking:
- Allow creation of
SegmentFileBoundary
- Allow creation of
RecordingMetadata
- Allow creation of
- Make dry run output more precise by avoiding the calculation of new start positions before the existing start position in the built-in policies.
- Introduced a new composite policy:
-
RetainMinOfPolicy
: combines sub-policies by retaining the minimum of its inner policies.
-
- Fixed a bug where
DeleteRecordingsAfterPeriodPolicy
gave the wrong retention instruction for live recordings. Thankfully, the purge would always fail, as the recording was live. A test has been added to prevent regressions.
-
Introduce a "dry run" mode, where the tool does not delete any data, but logs what it would have done. This can be useful for testing policies.
-
Change the default
controlResponseStreamId
we use from20
to20240712
to reduce the chances of collisions with other AeronArchive clients.
- Make it easier to configure the
controlReponseStreamId
on theAeronArchive.Context
that the DRR uses. - Avoid attempts to detach partial (i.e., not complete) segment files in
DeleteSegmentsAfterPeriodPolicy
if they have "expired".
- Renamed
DeleteAfterPeriodPolicy
to the more-specificDeleteRecordingsAfterPeriodPolicy
- Introduced new policies:
-
DeleteSegmentsAfterPeriodPolicy
: detaches and deletes recording segment files that have not been modified for some period of time. -
MatchingLogRecordingPolicy
: applies an inner policy to the Consensus Module's log recording.
-
Initial release with the following built-in data retention policies:
- Leaf policies:
-
RetainEnoughDataForRecoveryPolicy
: keeps the last N snapshots and the log following the earliest of these snapshots available for cluster node recovery. -
RetainLengthPolicy
: keeps at least the last N bytes of data. It will detach and delete recording segments before the last N bytes. -
DeleteAfterPeriodPolicy
: deletes recordings that stopped some period of time ago.
-
- Composite policies:
-
MatchingStreamsPolicy
: applies an inner policy to recordings that match a channel URI and/or stream identifier. -
RetainMaxOfPolicy
: safely combines policies, e.g., to retain X snapshots worth of cluster data and Y bytes of data per recording, by retaining the maximum of its inner policies. -
CascadePolicy
: accepts a sequence of policies. It will try each policy in turn and apply the first policy that "covers" a recording, i.e., that has an opinion on what the extent of the recording should be.
-