Skip to content

Conversation

@gvz
Copy link
Contributor

@gvz gvz commented Nov 19, 2025

when the parent directories name contains a "=" sign the code traverses upwards until no "=" is found, the parts behind the "=" are joined together with / to create a tree to make the data easy to select when a handling lots of data.

Apache hive format:
test
├── year=2021
│ ├── month=1
│ │ ├── file1.parquet
│ │ └── file2.parquet
│ └── month=2
│ └── file3.parquet

becomes 2021/1/ as the suggestion in the prefix dialog

@facontidavide
Copy link
Owner

this is nice, but as a directory like this can contain a LOT of files, I think It would be good to have a popup that display the number and size of the files that will be loaded and potentially a checkbox to select / deselect them

@gvz
Copy link
Contributor Author

gvz commented Nov 22, 2025

I think, I was not clear enough with my explanation.
This does not load all files in the parent directories automatically.
It is just changing the suggested prefixes from the filename to the hive structure path of the file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants