Share S3 client between CRR tasks #2642

Kerkesni · 2025-04-10T13:02:40Z

Share the same S3 clients (and their credentials) between multiple CRR tasks of the same site.

The ClientsManager class is an improved version of ClientManager that allows handling multiple clients with different auth configs. For now we still keep the ClientManager class to limit the scope of this PR, but code that uses it should be updated to use the newer class. This will be done in another PR.

Issue: BB-663

bert-e · 2025-04-10T13:02:44Z

Hello kerkesni,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2025-04-10T13:03:02Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

codecov · 2025-04-10T13:04:22Z

Codecov Report

Attention: Patch coverage is 97.67442% with 2 lines in your changes missing coverage. Please review.

Project coverage is 73.83%. Comparing base (774124a) to head (3e1ccf1).

Files with missing lines	Patch %	Lines
lib/clients/ClientsManager.js	96.92%	2 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
...sions/replication/queueProcessor/QueueProcessor.js	`74.63% <100.00%> (+1.69%)`	⬆️
extensions/replication/tasks/ReplicateObject.js	`91.94% <100.00%> (+0.11%)`	⬆️
lib/clients/ClientsManager.js	`96.92% <96.92%> (ø)`

... and 3 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	`75.40% <ø> (ø)`
Core Library	`80.03% <96.92%> (+0.27%)`	⬆️
Ingestion	`70.17% <ø> (ø)`
Lifecycle	`76.88% <ø> (ø)`
Oplog Populator	`85.06% <ø> (ø)`
Replication	`61.49% <100.00%> (+0.32%)`	⬆️
Bucket Scanner	`85.60% <ø> (ø)`

@@                 Coverage Diff                 @@
##           development/9.1    #2642      +/-   ##
===================================================
+ Coverage            73.63%   73.83%   +0.19%     
===================================================
  Files                  201      202       +1     
  Lines                13439    13516      +77     
===================================================
+ Hits                  9896     9979      +83     
+ Misses                3533     3527       -6     
  Partials                10       10

Flag	Coverage Δ
api:retry	`9.44% <0.00%> (-0.06%)`	⬇️
api:routes	`9.26% <0.00%> (-0.06%)`	⬇️
bucket-scanner	`85.60% <ø> (ø)`
ft_test:queuepopulator	`8.99% <0.00%> (-0.06%)`	⬇️
ingestion	`12.51% <0.00%> (-0.08%)`	⬇️
lib	`7.59% <0.00%> (-0.04%)`	⬇️
lifecycle	`18.88% <0.00%> (-0.12%)`	⬇️
notification	`1.05% <0.00%> (-0.01%)`	⬇️
replication	`18.73% <17.44%> (-0.08%)`	⬇️
unit	`49.89% <97.67%> (+0.40%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

create an improved version of the ClientManager class that allows storing multiple s3 clients with different auth configs. Issue: BB-663

When using AssumeRole auth, share the same destination S3 client in CRR throught the ClientsManager. Issue: BB-663

williamlardier · 2025-04-10T14:17:21Z

extensions/replication/queueProcessor/QueueProcessor.js

@@ -358,6 +362,14 @@ class QueueProcessor extends EventEmitter {
        }
    }

+    _setupClientsManager() {
+        if (this.destConfig.auth.type !== libConstants.authTypeAssumeRole) {


Maybe a comment here explaining why it is limited to assume role type would be great

williamlardier · 2025-04-10T14:21:36Z

lib/clients/ClientsManager.js

+const DELETE_INACTIVE_CREDENTIALS_INTERVAL = 1000 * 60 * 30; // 30m
+const MAX_INACTIVE_DURATION = 1000 * 60 * 60 * 2; // 2hr


If these are role sessions, they will likely expire before this timeout? Shall we link each set of creds with their expiration date (updated each time we refresh them)?

williamlardier · 2025-04-10T14:31:00Z

extensions/replication/tasks/ReplicateObject.js

-            const roleName = _extractRoleNameFromRole(targetRole);
-            this.clientManager = new ClientManager({
-                id: accountId,
+    _setupAssumeRoleDestClient(targetRole) {


Due to how errors are reported as null in case of issue, should we at least add some logs?

francoisferrand · 2025-05-27T07:35:30Z

extensions/replication/tasks/ReplicateObject.js

+    }
+
+    _setupDestClients(targetRole, log) {
+        this.destBackbeatHost = this.destHosts.pickHost();


is destBackbeatHost really a class attribute: should it not be just a parameter of _setupAssumeRoleDestClient ?

francoisferrand · 2025-05-27T07:37:18Z

extensions/replication/tasks/ReplicateObject.js

+        if (this.destConfig.auth.type === authTypeAssumeRole) {
+            this._setupAssumeRoleDestClient(targetRole);


since we have an asbtraction (clientManager), can't we just rely on it to create the client in any case?

i.e. maybe introduce an abstract class "ClientFactory", with 2 implementations...

note that it may be worth having 3 implementations:

CredentialsManager-based "AssumeRole" implementation

the simpler _createCredentials based implementation used in the 'else' branch of thsi functioj

CachedClientFactory adapter, which could be used on top of either implementation (as needed)

francoisferrand · 2025-05-27T07:40:57Z

lib/clients/ClientsManager.js

+        const client = this.s3Clients[clientId];
+
+        if (client) {
+            return client;


when we go through here, the credentials (credentials = this.credentialsManager.getCredentials) are not used --> useless to do it

francoisferrand · 2025-05-27T07:41:56Z

lib/clients/ClientsManager.js

+        const client = this.backbeatClients[clientId];
+
+        if (client) {
+            return client;


same, credentials not used in that case

francoisferrand · 2025-05-27T07:42:19Z

lib/clients/ClientsManager.js

+        const config = this._configs[clientId];
+        if (!config) {
+            return null;
+        }
+


useless, this would mean client is null

francoisferrand · 2025-05-27T07:51:38Z

lib/clients/ClientsManager.js

+     * @param {String} clientId - The client id.
+     * @return {AWS.S3} The S3 client instance to make requests with
+     */
+    getS3Client(clientId) {


the structure of getS3Client and getBackbeatClient is very redundant : probably should be refactored with a _getClient function, maybe something like

_getClient(clients /* map */, clientId /* string */, createClient /* func */) { const client = clients[clientId]; if (client) { return client; } const config = this._configs[clientId]; if (!config) { return null; } const credentials = this.credentialsManager.getCredentials({ id: clientId, accountId: config.accountId, stsConfig: config.stsConfig, authConfig: config.authConfig, }); if (credentials === null) { return null; } const client = createClient(config, credentials); this.s3Clients[clientId] = client; return client; }

scality deleted a comment from bert-e Apr 10, 2025

Kerkesni force-pushed the improvement/BB-663 branch from f02a2ff to 602390c Compare April 10, 2025 13:09

Kerkesni added 2 commits April 10, 2025 15:11

add the ClientsManager class

075b7a5

create an improved version of the ClientManager class that allows storing multiple s3 clients with different auth configs. Issue: BB-663

reuse the same BackbeatClient in CRR

3e1ccf1

When using AssumeRole auth, share the same destination S3 client in CRR throught the ClientsManager. Issue: BB-663

Kerkesni force-pushed the improvement/BB-663 branch from 602390c to 3e1ccf1 Compare April 10, 2025 13:11

williamlardier reviewed Apr 10, 2025

View reviewed changes

francoisferrand reviewed May 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Share S3 client between CRR tasks #2642

Share S3 client between CRR tasks #2642

Uh oh!

Kerkesni commented Apr 10, 2025

Uh oh!

bert-e commented Apr 10, 2025

Uh oh!

bert-e commented Apr 10, 2025

Uh oh!

codecov bot commented Apr 10, 2025 •

edited

Loading

Uh oh!

williamlardier Apr 10, 2025

Uh oh!

williamlardier Apr 10, 2025

Uh oh!

williamlardier Apr 10, 2025

Uh oh!

francoisferrand May 27, 2025

Uh oh!

francoisferrand May 27, 2025

Uh oh!

francoisferrand May 27, 2025

Uh oh!

francoisferrand May 27, 2025

Uh oh!

francoisferrand May 27, 2025

Uh oh!

francoisferrand May 27, 2025

Uh oh!

francoisferrand May 27, 2025

Uh oh!

Uh oh!

		const DELETE_INACTIVE_CREDENTIALS_INTERVAL = 1000 * 60 * 30; // 30m
		const MAX_INACTIVE_DURATION = 1000 * 60 * 60 * 2; // 2hr

		if (this.destConfig.auth.type === authTypeAssumeRole) {
		this._setupAssumeRoleDestClient(targetRole);

Share S3 client between CRR tasks #2642

Are you sure you want to change the base?

Share S3 client between CRR tasks #2642

Uh oh!

Conversation

Kerkesni commented Apr 10, 2025

Uh oh!

bert-e commented Apr 10, 2025

Hello kerkesni,

Uh oh!

bert-e commented Apr 10, 2025

Waiting for approval

Uh oh!

codecov bot commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Apr 10, 2025 •

edited

Loading