Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a safer interface to setPlanFnOutputs #1471

Open
ag-eitilt opened this issue Nov 21, 2023 · 0 comments
Open

Provide a safer interface to setPlanFnOutputs #1471

ag-eitilt opened this issue Nov 21, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@ag-eitilt
Copy link
Collaborator

Filtering out uninteresting files in the Plan/Job itself rather than only after getting an allOutputs : List Path has some promising implications for hashing optimization, database management, etc., but currently the only hook for that is FnOutputs: List String => List String which is a particularly dangerous function to use outside initially defining a runner. We should provide a OutputsFilter: String => Boolean entry point which makes the interface a bit less of a footgun.

Past discussions have talked about implementing this on top of the FnOutputs field, but if we fully separate it into a first-class member, we'd be able to leave FnOutputs as detecting "all files created by the Job" while OutputsFilter would be "all interesting files created by the Job", and the difference between the two provides a very easy determination of "supplemental files" which would need to be cleaned but not hashed/cached/etc.

(I know this keeps coming up in meetings, but I didn't see any issue tracking it.)

@ag-eitilt ag-eitilt added the enhancement New feature or request label Nov 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant