Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Arrow NativeFile and PythonFile for remote ORC storage #9377

Merged
merged 5 commits into from
Oct 7, 2021

Conversation

rjzamora
Copy link
Member

@rjzamora rjzamora commented Oct 5, 2021

This is a follow-up to #9304, and is more-or-less the ORC version of #9376

These changes will enable partial IO to behave "correctly" for cudf.read_orc from remote storage. Simpe multi-stripe file example:

# After this PR
%time gdf = cudf.read_orc(orc_path, stripes=[0], storage_options=storage_options)
CPU times: user 579 ms, sys: 166 ms, total: 744 ms
Wall time: 2.38 s

# Before this PR
%time gdf = cudf.read_orc(orc_path, stripes=[0], storage_options=storage_options)
CPU times: user 3.9 s, sys: 1.47 s, total: 5.37 s
Wall time: 8.5 s

@rjzamora rjzamora added 3 - Ready for Review Ready for review by team Python Affects Python cuDF API. Cython Performance Performance related issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Oct 5, 2021
@rjzamora rjzamora requested a review from a team as a code owner October 5, 2021 16:28
@rjzamora rjzamora self-assigned this Oct 5, 2021
@codecov
Copy link

codecov bot commented Oct 5, 2021

Codecov Report

Merging #9377 (80eeba3) into branch-21.12 (ab4bfaa) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

❗ Current head 80eeba3 differs from pull request most recent head 01d06bb. Consider uploading reports for the commit 01d06bb to get more accurate results
Impacted file tree graph

@@               Coverage Diff                @@
##           branch-21.12    #9377      +/-   ##
================================================
- Coverage         10.79%   10.74%   -0.05%     
================================================
  Files               116      116              
  Lines             18869    19081     +212     
================================================
+ Hits               2036     2051      +15     
- Misses            16833    17030     +197     
Impacted Files Coverage Δ
python/cudf/cudf/__init__.py 0.00% <0.00%> (ø)
python/cudf/cudf/_lib/__init__.py 0.00% <ø> (ø)
python/cudf/cudf/core/_base_index.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/categorical.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/column.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/datetime.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/lists.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/numerical.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/string.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/column/struct.py 0.00% <0.00%> (ø)
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 88eefe5...01d06bb. Read the comment docs.

Copy link
Member

@charlesbluca charlesbluca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

python/cudf/cudf/_lib/orc.pyx Outdated Show resolved Hide resolved
Co-authored-by: Charles Blackmon-Luca <[email protected]>
@rjzamora
Copy link
Member Author

rjzamora commented Oct 7, 2021

@gpucibot merge

@rapids-bot rapids-bot bot merged commit f4ff454 into rapidsai:branch-21.12 Oct 7, 2021
@rjzamora rjzamora deleted the orc-pythonfile branch October 7, 2021 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Performance Performance related issue Python Affects Python cuDF API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants