Skip to content

(feat) Optimizer & Memo Full Logical -> Logical #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
May 13, 2025
Merged

Conversation

AlSchlo
Copy link
Collaborator

@AlSchlo AlSchlo commented May 7, 2025

Implements the in-memory memo for logical -> logical, as well as the optimizer task graph, and the support of * stored expressions in the compiler.

All features are tested accordingly, except the optimizer itself which will be postponed in a future PR.

@AlSchlo AlSchlo changed the title Reorg src dir for optimizer (feat) Optimizer Full Logical -> Logical May 7, 2025
@AlSchlo AlSchlo changed the title (feat) Optimizer Full Logical -> Logical (feat) Optimizer & Memo Full Logical -> Logical May 7, 2025
@codecov-commenter
Copy link

codecov-commenter commented May 7, 2025

Codecov Report

Attention: Patch coverage is 41.32771% with 1529 lines in your changes missing coverage. Please review.

Project coverage is 82.6%. Comparing base (ad02315) to head (c075611).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
optd/src/optimizer/tasks/launch.rs 0.0% 207 Missing ⚠️
optd/src/optimizer/jobs/execute.rs 0.0% 194 Missing ⚠️
optd/src/optimizer/tasks/manage.rs 0.0% 179 Missing ⚠️
optd/src/optimizer/merge/helpers.rs 0.0% 160 Missing ⚠️
optd/src/optimizer/handlers.rs 0.0% 145 Missing ⚠️
optd/src/optimizer/memo_io/egest.rs 0.0% 124 Missing ⚠️
optd/src/optimizer/memo_io/ingest.rs 0.0% 119 Missing ⚠️
optd/src/optimizer/mod.rs 0.0% 117 Missing ⚠️
optd/src/optimizer/jobs/manage.rs 0.0% 46 Missing ⚠️
optd/src/optimizer/tasks/delete.rs 0.0% 46 Missing ⚠️
... and 13 more
Additional details and impacted files
Files with missing lines Coverage Δ
optd/src/dsl/analyzer/hir/mod.rs 78.7% <100.0%> (+1.1%) ⬆️
optd/src/dsl/analyzer/type_checks/glb.rs 96.6% <100.0%> (ø)
optd/src/dsl/analyzer/type_checks/lub.rs 98.2% <100.0%> (ø)
optd/src/dsl/analyzer/type_checks/subtype.rs 99.7% <100.0%> (ø)
optd/src/dsl/engine/eval/core.rs 98.9% <100.0%> (+<0.1%) ⬆️
optd/src/dsl/engine/eval/expr.rs 98.3% <100.0%> (+<0.1%) ⬆️
optd/src/dsl/engine/eval/match.rs 99.3% <100.0%> (+<0.1%) ⬆️
optd/src/dsl/engine/eval/operator.rs 100.0% <100.0%> (ø)
optd/src/dsl/engine/mod.rs 67.0% <80.0%> (+1.2%) ⬆️
optd/src/memo/memory/union_find.rs 98.6% <98.6%> (ø)
... and 21 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@connortsui20 connortsui20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice job! I like the design here and the fact that generally different ideas are separated along different methods and types. I think the new group ID for newly merged groups is also generally a good idea now - before when we were trying to think about parallelism it wasn't ideal but since that is out the window we don't really care about that anymore.

I suggested a few things, but the only thing that I guess is worth discussing is the error handling on the memo table. Basically, what should happen if something goes wrong? See my comments above.

AlSchlo and others added 11 commits May 12, 2025 18:07
## Problem

Because the memo table implementations are likely going to have very
different error conditions, we would ideally like the implementors of
the memo table traits to define their errors instead of us.

For example, a persistent memo table on disk will want to return I/O
errors, but an in-memory memo table should be infallible. A memo table
as a service over the network might have completely different errors,
and a parallel memo table could have different behavior as well.

## Summary of changes

This PR modifies the `memo` module to use a new `MemoBase` trait that
allows the implementors to define an associated error type.

```rust
/// Base trait defining a shared implemention-defined error type for all memo-related traits.
pub trait MemoBase {
    /// The error type used throughout all memo-related traits.
    type MemoError: Debug;
}
```

The reason we need this is because the memo traits have been effectively
divided into 3 subtraits, and we cannot specify 1 associated type for 3
different traits.

---

Note that the error type for the in-memory memo table is `Infallible`.

```rust
/// The never type. Used for ensuring that the in-memory memo table never raises an error.
#[derive(Debug)]
pub enum Infallible {}

/// This defines the error type for the in-memory implementation of the memo table.
impl MemoBase for MemoryMemo {
    /// In-memory operations cannot fail.
    type MemoError = Infallible;
}
```

This could change in the future, but at least for now it is not really
clear what a user of the in-memory table would do if it encountered a
broken invariant (a group does not exist, for example). So this PR
simply makes all of those error cases a panic / crash. This can easily
be changed in the future.

# TODO

- [x] Figure out how to error handle in the optimizer code
- [x] Refactor how the `optimizer` module handles errors with
`OptimizerError`, since `MemoError` used to be the only variant
- [x] Clean up lints
AlSchlo pushed a commit that referenced this pull request May 13, 2025
## Summary of changes

This PR moves around the modules in the main branch in order to help
merge #101 more easily.
Copy link
Member

@connortsui20 connortsui20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I only looked at the things related to the memo but I think this is fine to merge and clean up later.

@AlSchlo AlSchlo merged commit ed681b5 into main May 13, 2025
12 checks passed
@AlSchlo AlSchlo deleted the alexis/optimizer branch May 13, 2025 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants