feat: Single Active Replication

[bupd]

Single Active Replication per replication policy

Proposal: goharbor/community#256

Summary

This PR addresses a long-standing issue where overlapping replications of the same policies can occur in Harbor, leading to unnecessary resource consumption and poor performance. By introducing a "Single Active Replication" checkbox in the replication policy, it ensures that replication tasks for the same policy do not run if there is already a replication running for the same policy, preventing bandwidth overload and queue backups, especially for large artifacts.

Similar Issues

• Replication job duplication #19937
• Maximum parallel replications for pull #20532
• Replication: avoid overlapping replication jobs #17269
• Replication taking longer each time #16656

Related Issues

• Replication to second harbor is extremely slow #15415
• replication job failed,but do not re-scheduled #7842

Why do we need this

1. Users have for long requesting this feature to have single replication execution per policy.
2. It makes no sense to run replication in parallel for same replication policy.

Changes Made

• Added Single active replication Checkbox in Replication UI.
• Updated Replication Policy Schema.
• Added single_active_replication column in replication_policy table in sql.

Screenshots

Observation on Replication Performance with and without Feature

Speed Test Results:

Upload Speed: 9.50 Mbps (Data Used: 4.3 MB)
Latency: 37.96 ms (Jitter: 1.76 ms, Min: 25.11 ms, Max: 43.32 ms)
Packet Loss: 0.0%
Result URL: Speedtest Result

Images to Replicate: Variations of docker.io/vad1mo/1gb-random-file

Workflow 1 (with Feature):

1 x image (512 MB)
3 x images (~1.5 GB)
From: reg1.bupd.xyz to reg2.bupd.xyz
Replication: Normal
Started: 2:27 PM
Completed: 2:49 PM
Bandwidth Used: 1.42 GB
Theoretical Time: 22.5 minutes for 1.5 GB
Actual Time: 22 minutes (No packet loss or bandwidth issues)

Insight

Replication with the feature enabled completed within the expected timeframe, showing stable upload speed and no packet loss.

Workflow 2 (without Feature):

Started: 2:55 PM
Name: Destroyer
Bandwidth Used: 13+ GB
Result: Failed
Time Taken: ~4+hrs

Status of Instance (no longer functioning)

Todo

• Add Tests

Please indicate you've done the following:

• Well Written Title and Summary of the PR
• Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
• Accepted the DCO. Commits without the DCO will delay acceptance.
• Made sure tests are passing and test coverage is added if needed.
• Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

Fix #19937

[codecov]

Codecov Report

❌ Patch coverage is 41.02564% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.88%. Comparing base (c8c11b4) to head (3d6d810).
⚠️ Report is 532 commits behind head on main.

Files with missing lines	Patch %	Lines
src/server/v2.0/handler/replication.go	0.00%	13 Missing ⚠️
src/controller/replication/execution.go	70.00%	4 Missing and 2 partials ⚠️
...ion/create-edit-rule/create-edit-rule.component.ts	0.00%	2 Missing ⚠️
...v/replication/replication/replication.component.ts	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #21347       +/-   ##
===========================================
+ Coverage   45.36%   65.88%   +20.51%     
===========================================
  Files         244     1072      +828     
  Lines       13333   115932   +102599     
  Branches     2719     2927      +208     
===========================================
+ Hits         6049    76384    +70335     
- Misses       6983    35315    +28332     
- Partials      301     4233     +3932     

Flag	Coverage Δ
unittests	65.88% <41.02%> (+20.51%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/controller/replication/model/model.go	48.26% <100.00%> (ø)
src/pkg/replication/model/model.go	100.00% <ø> (ø)
...ion/create-edit-rule/create-edit-rule.component.ts	36.03% <0.00%> (+0.31%)	⬆️
...v/replication/replication/replication.component.ts	22.60% <0.00%> (+0.14%)	⬆️
src/controller/replication/execution.go	69.03% <70.00%> (ø)
src/server/v2.0/handler/replication.go	0.00% <0.00%> (ø)

... and 980 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[wy65701436]

Thanks, @bupd, for your contribution! I believe this is a valuable new feature for Harbor that should follow the proposal process.

There are several questions we need to briefly discuss:

• What happens if the execution gets stuck in the 'running' status?
• How do we determine the relationship between the replication policy and the execution instance — by ID? And what happens if the policy content is updated after the execution begins?
• During the replication job (at least in Harbor-to-Harbor replication), Harbor skips artifacts that have already been successfully replicated. So, we should consider how much additional benefit this feature will bring.

[tpoxa]

Thanks, @bupd, for your contribution! I believe this is a valuable new feature for Harbor that should follow the proposal process.

There are several questions we need to briefly discuss:

• What happens if the execution gets stuck in the 'running' status?

We were planning to use JobServiceMonitorClient to get the current state of job observations. As long observation says it's running - a new execution will be skipped. I believe we should fix the problem with stuck separately. What are your thoughts about that?

• How do we determine the relationship between the replication policy and the execution instance — by ID? And what happens if the policy content is updated after the execution begins?

Indeed PolicyID was never needed before. We can add it anyway to the job arguments so we can scan observations, parse job arguments and match policyID with the current.

• During the replication job (at least in Harbor-to-Harbor replication), Harbor skips artifacts that have already been successfully replicated. So, we should consider how much additional benefit this feature will bring.

I believe even checking an artefact for existence produces an additonal HTTP request. So, we still adding some extra efficiency.

[Vad1mo]

• During the replication job (at least in Harbor-to-Harbor replication), Harbor skips artifacts that have already been successfully replicated. So, we should consider how much additional benefit this feature will bring.

Regarding that statement. there is a fundamental deficit with harbor replications, when the layer is in the copy process, new replication will try to copy the same blob again. If you have a slow connection like DSL the same blob will be copied over and over again. This results in the domino effect. Each new replication halves the throughput, resulting in slower transfers, and because all replications run in parallel, the same effect happens with the next blob/layer.

We have made an experiment with:

• Copying 1 GiB image
• Limit transfer speed 1 Mbit, per replication
  • Please note that we set the 1Mib per replication. On a real DSL, 3G connection, the total bandwidth (1 Mbit) would not be shared across all replications, so each replication would get 1/n of the bandwidth.
  • So the observed time needs to be multiplied by n.
• Replication every minute
• Replication time with "Prevent Overlapping Replications" Enabled ->: 22 min
• Replications time with "Prevent Overlapping Replications" Disabled -> : Failed after 4+ hours

[bupd]

Hello @wy65701436

The below are my observations.

• What happens if the execution gets stuck in the 'running' status?

While single active replication is checked. Replication executions per policy will become to atmost one. So if an execution is already running. other scheduled executions will be defered.

• How do we determine the relationship between the replication policy and the execution instance — by ID? And what happens if the policy content is updated after the execution begins?

1. each execution will have a replication policyID attached to it. so on executing the new jobs we check if there is any jobs that are already running with same execution policyID. if we find any jobs that are in progress. we will defer the executions until the previous job completes.
2. By design, once harbor replication execution starts the policy content is fixed. and changes made to replication policy will only reflect in upcoming replication executions.

• During the replication job (at least in Harbor-to-Harbor replication), Harbor skips artifacts that have already been successfully replicated. So, we should consider how much additional benefit this feature will bring.

Yes exactly, Harbor only skips artifacts that have already been successfully replicated.
So in typical scenario harbor will try copying the same layers with different jobs pushing the same layer.

In the below scenario I tried replicating three images each of size: 512mb. but the resulting repository quota
resulting something similar like this.

Project Quota

Where the quota used should only be less than 1.5gb.

4.77GB is used here for project due to replication jobs copying the same layer simultaneously.
this clearly shows the quota consumed is greater than the images needed to be replicated.

The below is the overall quota

Attaching here for reference.

Thanks

You can check the two instances used on this experiment: reg1.bupd.xyz & reg2.bupd.xyz

[bupd]

Observation on Replication Performance with and without Feature

Speed Test Results:

Upload Speed: 9.50 Mbps (Data Used: 4.3 MB)
Latency: 37.96 ms (Jitter: 1.76 ms, Min: 25.11 ms, Max: 43.32 ms)
Packet Loss: 0.0%
Result URL: Speedtest Result

Images to Replicate: Variations of docker.io/vad1mo/1gb-random-file

Workflow 1 (with Feature):

1 x image (512 MB)
3 x images (~1.5 GB)
From: reg1.bupd.xyz to reg2.bupd.xyz
Replication: Normal
Started: 2:27 PM
Completed: 2:49 PM
Bandwidth Used: 1.42 GB
Theoretical Time: 22.5 minutes for 1.5 GB
Actual Time: 22 minutes (No packet loss or bandwidth issues)

Insight

Replication with the feature enabled completed within the expected timeframe, showing stable upload speed and no packet loss.

Workflow 2 (without Feature):

Started: 2:55 PM
Name: Destroyer
Bandwidth Used: 13+ GB
Result: Failed
Time Taken: ~4+hrs (failed to complete)

Status of Instance

Insight

The replication process failed without the feature enabled, highlighting that the feature is likely essential for successful image transfers between reg1.bupd.xyz and reg2.bupd.xyz. This indicates that the feature could be critical for ensuring stable uploads and completing transfers within the expected time.

Conclusion:

This observation suggests that this feature is necessary for reliable replication between registries. The data transfer rates and lack of packet loss when the feature is used make it an essential component for stable image replication.

[bupd]

you can say a workaround here:

please set scheduled replication interval more than enough time it needs. to be safe.
-- or --
The cron string is 0 */2 * * * *, it means that the replication is start every 2 minutes, you should always keep the schedule interval longer than a single job complete time.
#20532 (comment)

The workaround of setting a longer replication interval, like once a day, fails to address the need for timely synchronization across registries. For users who rely on Harbor to maintain identical registries at different locations, frequent replication (e.g., every 5 minutes) is necessary to ensure minimal discrepancies between registries. By suggesting a longer interval, users may end up with outdated or inconsistent images, undermining the core functionality of replication.

Thanks.

[reasonerjt]

lgtm

[chlins]

lgtm