[ENH] Only load blocks for updated records

[Sicheng-Pan]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Optimizes the materialize logic by moving record segment block load out into materialize operator, and selectively load only the updated records.
• New functionality
  • N/A

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?

Observability plan

What is the plan to instrument and monitor this change?

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[Sicheng-Pan]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [ENH] Only load blocks for updated records #6007 👈 (View in Graphite)
• [ENH] Load blocks given keys under P0 priority #6006
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[propel-code-bot]

Optimize selective record block loading during materialization

The patch moves block prefetching out of materialize_logs and into MaterializeLogOperator, where it now calls load_id_to_data only for records whose final operation is MaterializedLogOperation::UpdateExisting. This keeps the materializer result computation unchanged while reducing unnecessary block loads for records that will be hydrated solely from log data.

Key Changes

• Invoke RecordSegmentReader::load_id_to_data from MaterializeLogOperator::run and filter the prefetch list to logs with MaterializedLogOperation::UpdateExisting.
• Remove the unconditional load_id_to_data call for all existing offsets inside materialize_logs.

Affected Areas

• rust/worker/src/execution/operators/materialize_logs.rs
• rust/segment/src/types.rs

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[Performance] The current filtering for pre-loading data is too restrictive. It only considers UpdateExisting operations. However, records with DeleteExisting and OverwriteExisting operations also require their original data from the record segment to correctly calculate the logical size delta and for segment writers to process them.

Without pre-loading, the data for these records will be fetched individually within the hydrate call inside the loop, which negates some of the performance benefit of this optimization.

To fix this, we should pre-load data for all operations on existing records. Since Initial operations are already filtered out by materialize_logs, this means we should load for all operations except AddNew.

Context for Agents

The current filtering for pre-loading data is too restrictive. It only considers `UpdateExisting` operations. However, records with `DeleteExisting` and `OverwriteExisting` operations also require their original data from the record segment to correctly calculate the logical size delta and for segment writers to process them.

Without pre-loading, the data for these records will be fetched individually within the `hydrate` call inside the loop, which negates some of the performance benefit of this optimization.

To fix this, we should pre-load data for all operations on existing records. Since `Initial` operations are already filtered out by `materialize_logs`, this means we should load for all operations except `AddNew`.

File: rust/worker/src/execution/operators/materialize_logs.rs
Line: 87

[Sicheng-Pan]

Inspected downstream changes, may need more optimization to make this work. Will close for now