Skip to content

Conversation

keith-turner
Copy link
Contributor

Compaction selection functionality that opened files was not implemented because it opens user files in the manager. This change implements the functionality inorder to get integration test passing. There is already an open issue about finding a better way to do this. See #3526

Compaction selection functionality that opened files was not implemented
because it opens user files in the manager.  This change implements the
functionality inorder to get integration test passing.  There is already
an open issue about finding a better way to do this. See apache#3526
@keith-turner
Copy link
Contributor Author

This PR along with #3785 and #3784 are all part of an effort to get mvn verify -Psunny working in the elasticity branch.

@dlmarion
Copy link
Contributor

dlmarion commented Oct 3, 2023

Would this be another task type for #3801 ?

@keith-turner
Copy link
Contributor Author

Would this be another task type for #3801 ?

Yeah, I think it may. It will be similar to computing splits. Wondering how all of these should return data. If the manager stores results in memory then it has to store the data until the FATE operation runs again and is ready to consume it. Thinking it could write the result to the metadata table instead of storing it in manager memory, maybe something like the following.

  1. Fate op needs computation done for tablet and queues a task and then exits (fate will run it later)
  2. The task eventually runs on a worker, computes the result, and then writes the result to the tablets row in the metadata table.
  3. Fate step runs again and sees the result of the computation in the metadata table and uses it

@dlmarion
Copy link
Contributor

dlmarion commented Oct 4, 2023

I wonder if we should get #3801 where it needs to be and merged, then implement this in that framework vs merging this to get sunny working just to have to back it out later.

@keith-turner
Copy link
Contributor Author

I wonder if we should get #3801 where it needs to be and merged, then implement this in that framework vs merging this to get sunny working just to have to back it out later.

Its a small change. I would like to get the verify sunny build working. Would like to also look into expanding what is in the sunny tests to make sure all major features are well covered (ingest, scan, compact, split, merge, bulk import, etc).

@keith-turner keith-turner merged commit 164c85c into apache:elasticity Oct 6, 2023
@ctubbsii ctubbsii added this to the 4.0.0 milestone Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants