-
Notifications
You must be signed in to change notification settings - Fork 338
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
The spec defines metadata columns, one of which is the _file column. This column seems instrumental for write operations positional deletes in future, as well as for having some ways for users to identify rows (combination of filepath + position)
Describe the solution you'd like
The solution should allow building a TableScan with option to return metadata columns (in this case _file). When that happens, the library should return batches that include this extra column. The column should be encoded in such a way that it doesn't add much memory overhead. Since arrow doesn't support constant vectors, perhaps RLE could be used.
Willingness to contribute
I can contribute to this feature independently
liurenjie1024
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request