You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Filtering out uninteresting files in the Plan/Job itself rather than only after getting an allOutputs : List Path has some promising implications for hashing optimization, database management, etc., but currently the only hook for that is FnOutputs: List String => List String which is a particularly dangerous function to use outside initially defining a runner. We should provide a OutputsFilter: String => Boolean entry point which makes the interface a bit less of a footgun.
Past discussions have talked about implementing this on top of the FnOutputs field, but if we fully separate it into a first-class member, we'd be able to leave FnOutputs as detecting "all files created by the Job" while OutputsFilter would be "all interesting files created by the Job", and the difference between the two provides a very easy determination of "supplemental files" which would need to be cleaned but not hashed/cached/etc.
(I know this keeps coming up in meetings, but I didn't see any issue tracking it.)
The text was updated successfully, but these errors were encountered:
Filtering out uninteresting files in the
Plan
/Job
itself rather than only after getting anallOutputs : List Path
has some promising implications for hashing optimization, database management, etc., but currently the only hook for that isFnOutputs: List String => List String
which is a particularly dangerous function to use outside initially defining a runner. We should provide aOutputsFilter: String => Boolean
entry point which makes the interface a bit less of a footgun.Past discussions have talked about implementing this on top of the
FnOutputs
field, but if we fully separate it into a first-class member, we'd be able to leaveFnOutputs
as detecting "all files created by theJob
" whileOutputsFilter
would be "all interesting files created by theJob
", and the difference between the two provides a very easy determination of "supplemental files" which would need to be cleaned but not hashed/cached/etc.(I know this keeps coming up in meetings, but I didn't see any issue tracking it.)
The text was updated successfully, but these errors were encountered: