-
Notifications
You must be signed in to change notification settings - Fork 3
feat(loader): Add dataframe/huggingface specific classes #80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great PR! The main improvement is that Hugging Face data can now be used directly in the pipelines without converting it to a CSV first—this removes redundancy, which is excellent.
Just one minor fix: the docs/copy_of_examples
folder should be removed, as it seems to have been committed by mistake. Apart from that, everything looks ready to merge.
If @fabiofelix did change the examples he needs to push the copy_of_examples in docs otherwise the "notebooks" are not forwarded to the documentation; just in case ✅ Let me have a Quick Look at the PR I have 5" before merging guys please Cheers |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Amazing PR @fabiofelix ! Nothing of a big deal to change just to convince me this is the right way / expected, let me know!
Cheers
examples/3-Case-Studies/4-Paris-Remarkable-Trees/1-Paris_Remarquable_Trees_Pipeline.ipynb
Show resolved
Hide resolved
examples/3-Case-Studies/4-Paris-Remarkable-Trees/1-Paris_Remarquable_Trees_Pipeline.ipynb
Show resolved
Hide resolved
...es/3-Case-Studies/4-Paris-Remarkable-Trees/2-Paris_Remarquable_Trees_Advanced_Pipeline.ipynb
Show resolved
Hide resolved
...es/3-Case-Studies/4-Paris-Remarkable-Trees/2-Paris_Remarquable_Trees_Advanced_Pipeline.ipynb
Show resolved
Hide resolved
...es/3-Case-Studies/4-Paris-Remarkable-Trees/2-Paris_Remarquable_Trees_Advanced_Pipeline.ipynb
Show resolved
Hide resolved
...es/3-Case-Studies/4-Paris-Remarkable-Trees/2-Paris_Remarquable_Trees_Advanced_Pipeline.ipynb
Show resolved
Hide resolved
...of_examples/3-Case-Studies/4-Paris-Remarkable-Trees/1-Paris_Remarquable_Trees_Pipeline.ipynb
Show resolved
Hide resolved
b293097
to
8fdb3a9
Compare
8fdb3a9
to
1fe8146
Compare
49a277a
to
a084d59
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Brilliant PR! Approving on my side of the table! Thanks for this great one @fabiofelix!
Next time, let's try to have a separate commit when you are changing things a little outside the scope of the PR, such as the viz. color palettes in the examples, so that this commit can have a description explaining why the change was made to avoid confusion, please.
I'll let @soniacq merge once she approves the PR 🫡
Cheers!
a084d59
to
311fdf5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, Fabio, for addressing all the comments.
The following PR is about:
It unifies
LoaderBase
interface by:load_data_from_file
withload
methodFileLoader
class that inheritsLoaderBase
. Therefore,CSVLoader
,ParquetLoader
, andShapefileLoader
inherit this classDataFrameLoader
andHuggingFaceLoader
, which inheritLoaderBase
, and moving specific codes fromLoaderFactory
.LoaderFactory
code, mainlybuild
andload
methods.Now, it is possible to use HuggingFace (or dataframe) in a
pipeline
in the same way as file loaders, calling the factory methodbuild
.Before, the user should've retrieved data from HuggingFace, saved it as a file, and used a file loader with the
pipeline
.📚 Documentation preview 📚: https://UrbanMapper--80.org.readthedocs.build/en/80/