Thawing implementation in the new logic #28

felipeji · 2025-06-13T02:53:11Z

This PR addresses Issue #13 and introduces a parallel thawing implementation to support copying objects stored in cold storage.

Cold objects are now identified and split into two copysets based on size: smallThaw and largeThaw (just as regular, hot data based in the 5 MiB threshold ). Each copyset is processed independently through its own thaw + copy map chain. Once thawed, files are copied using either Lambda (for small files) or Fargate (for large files), same that hot data.

This is implemented via two new constructs:

thaw-small-copy-map-construct.ts
thaw-large-copy-map-construct.ts

Each of these chains together thaw-objects-map-construct with the corresponding copy construct (small-objects-copy-map-construct or large-objects-copy-map-construct). These thawing chains are integrated into steps-s3-copy-construct.ts, running in parallel with hot data copiers.

In addition to the two new constructs, six other files were updated to integrate thawing and fix some issues encountered during implementation, as detailed below:

dev-testsuite/test-e2e-thawing.ts : the test timeout was increased by 2 minutes. Some thawing runs were cutting it close or failing under the original 5-minute limit.
lambda/coordinate-copy-lambda.ts new copy sets (smallThaw and largeThaw) are now created based on whether the object’s storage class is in COLD_STORAGE_CLASSES, and whether its size is below or above 5 MiB.
Minor TypeScript-related refactors:

Switched to ESM-style import for path/posix
Added non-null assertions (!) where needed to satisfy the compiler

(These two were done mostly to silence TS warnings— please, review it.)

small-objects-copy-map-construct.ts made the construct reusable by allowing lambdaStateName to be passed in rather than hardcoding it. There's still a bit of mixing between dynamic and hardcoded names—I might clean that up further in a follow-up ?

4&5. thaw-objects-lambda-step-construct.ts & thaw-objects-map-construct.ts: refactored to support reusable thawing logic.

Key changes:

Introduced dynamic naming for the lambdaFunction
Now using S3CsvDistributedMap instead of S3JsonlDistributedMap, since we're now reading thaw instructions directly from JSONL (via the copySets.smallThaw / largeThaw input)

steps-s3-copy-construct.ts: Integrated calls to the new thawing constructs using the same naming conventions and parallel execution logic as for hot data copiers.
NOTE: I’m manually granting invoke permissions to the thawing Lambdas (line ~350). It works, but I’m not entirely sure it’s the cleanest/correct way to do it.

…da.ts

andrewpatto · 2025-06-17T05:33:57Z

packages/steps-s3-copy/src/lib/thaw-large-copy-map-construct.ts

+      inputPath: props.inputPath,
+      mapStateName: id,
+    });
+


What would be the ramifications if one thaw completed quickly - and the others took 48 hours?

Might the quick thaw "re-expire"??

I thought potentially for (large or small) thawing we would have only one distributed map - but that within that map would be a single chain for each object doing the thaw->wait->copy loop

I mean this is fine - and is effectively what we had before in that we separated out the thawing map from the copying map. But I wonder if the other is more efficient/neater?

…r reuse. Refactored file names removing "object" from for consistency and improved readability.

…small object thawing still needs refactor to match this refactor.

…h small and large object copy constructs. Cleaned up and unified naming conventions across all related constructs for consistency.

felipeji · 2025-06-27T01:35:53Z

The previous implementation of thawing was not optimal, as it performed all thawing operations before initiating the copy tasks instead of in the same map, which is inefficiency.

To address this, I’ve moved the thawing logic into the copy map constructs themselves—SmallObjectsCopyMapConstruct and CopyMapConstruct, so that thawing is now triggered via the thaw flag when they are instantiated. This new approach allows each object to be thawed just before it is copied.

The logic is now implemented in:

packages/steps-s3-copy/src/lib/copy-map-construct.ts
packages/steps-s3-copy/src/lib/small-copy-map-construct.ts

So, as a result, the following thawing-specific constructs are no longer needed and have been removed:

packages/steps-s3-copy/src/lib/thaw-large-copy-map-construct.ts
packages/steps-s3-copy/src/lib/thaw-objects-map-construct.ts
packages/steps-s3-copy/src/lib/thaw-small-copy-map-construct.ts

Other changes include:

In packages/steps-s3-copy/lambda/can-read-objects-lambda/can-read-objects-lambda.ts, the original event is now passed downstream so that it remains available to the copy step after thawing.

In packages/steps-s3-copy/src/lib/thaw-lambda-step-construct.ts, I added outputPath: "$.Payload" to handling of InputPath in downstream copy steps.

I also cleaned up the state and step names for clarity, while still using CamelCase for naming. We could consider setting more human-readable names for those steps in the future (?)

andrewpatto · 2025-06-27T02:10:48Z

packages/steps-s3-copy/lambda/can-read-objects-lambda/can-read-objects-lambda.ts

-  return {};
+  }
+  // Return original input so downstream steps (e.g. ECS copy) get expected fields
+  return event;


Just as a note - you can also pass through to downstream steps with config in the Steps. As in - you can ignore the the return value of the lambda and have Steps itself pass through values.

No need to change here - is really possibly to do with us progressing to full JSONata at some point

I need to learn more on this. I was dealing quit a bit with this, and was the best solution I found (or at least the one I actually understood) cos the copy step brokes when trying to pass the inputs needed for thawing. I’m happy to work on trying the full implementation JSONata, so I can learn.

andrewpatto · 2025-06-27T02:11:59Z

packages/steps-s3-copy/src/lib/copy-map-construct.ts

+    } else {
+      entryState = delayStep;
+    }
+


andrewpatto · 2025-06-27T02:13:51Z

packages/steps-s3-copy/src/lib/copy-map-construct.ts

@@ -42,6 +43,9 @@ type Props = {

  readonly taskDefinition: TaskDefinition;
  readonly containerDefinition: ContainerDefinition;
+
+  readonly thaw?: boolean;


I would clarfiy the name of this - to make it clear that this thaw changes the Construct and is not dynamic behaviour.

So change it to be a verb about the construct. Maybe addThawStep?

andrewpatto · 2025-06-27T02:15:40Z

packages/steps-s3-copy/src/lib/copy-map-construct.ts

+        intelligentTieringDeepArchiveThawSpeed: props.aggressiveTimes
+          ? "Standard"
+          : "Bulk",
+      },


I will put in a new issue - so not for this PR - but we will eventually want these parameters to be allowed to be specified by the invoker of the copy (so possibly bringing this entire data structure up to the Steps invoke parameter level)

Yeah, definitely. Just keep it hard coded as it was but would be nice to having it as accessible params.

Felipe Jimenez-Ibarra added 7 commits June 6, 2025 10:24

define new copy set for small items requiring thawing

46de78a

dynamic naming for small object copy

85dc52c

first working version of thawing for small files

46c59d0

first fully working thawing implementation

193f5ac

rename thawing constructs and classes for clarity and consistency

e361a69

resolve TypeScript warnings and import issues in coordinate-copy-lamb…

53a3ca4

…da.ts

add thaw-copy constructs some comments

478e0b4

felipeji requested a review from andrewpatto June 13, 2025 02:53

andrewpatto reviewed Jun 17, 2025

View reviewed changes

Felipe Jimenez-Ibarra added 4 commits June 23, 2025 13:56

thaw and copy small objects in chained map step

7f6946b

SmallObjectsCopyLambdaStepConstruct moved to a dedicated construct fo…

678f643

…r reuse. Refactored file names removing "object" from for consistency and improved readability.

Implement thawing step in CopyMapConstruct triggered by thaw flag; …

0394fe5

…small object thawing still needs refactor to match this refactor.

Completed refactor of thawing logic triggered by the thaw flag in bot…

30db464

…h small and large object copy constructs. Cleaned up and unified naming conventions across all related constructs for consistency.

andrewpatto reviewed Jun 27, 2025

View reviewed changes

andrewpatto approved these changes Jun 27, 2025

View reviewed changes

Rename thaw flag to addThawStep for clarity on construct behavior.

27476fa

felipeji merged commit 70e196c into main Jun 30, 2025
6 checks passed

felipeji deleted the thawing-new-logic branch June 30, 2025 01:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Thawing implementation in the new logic #28

Thawing implementation in the new logic #28

Uh oh!

felipeji commented Jun 13, 2025

Uh oh!

andrewpatto Jun 17, 2025

Uh oh!

andrewpatto Jun 17, 2025

Uh oh!

andrewpatto Jun 17, 2025

Uh oh!

felipeji commented Jun 27, 2025

Uh oh!

andrewpatto Jun 27, 2025

Uh oh!

felipeji Jun 27, 2025

Uh oh!

andrewpatto Jun 27, 2025

Uh oh!

andrewpatto Jun 27, 2025

Uh oh!

felipeji Jun 27, 2025

Uh oh!

andrewpatto Jun 27, 2025

Uh oh!

felipeji Jun 27, 2025

Uh oh!

Uh oh!

Uh oh!

Thawing implementation in the new logic #28

Thawing implementation in the new logic #28

Uh oh!

Conversation

felipeji commented Jun 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felipeji commented Jun 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!