Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add initial Arrow PyCapsule support #517

Merged
merged 23 commits into from
Oct 15, 2024
Merged

Add initial Arrow PyCapsule support #517

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
9a09bdf
Add initial Arrow PyCapsule support
jonmmease Oct 12, 2024
bbfb563
Bump python in actions to 3.11
jonmmease Oct 12, 2024
c5652dc
fmt
jonmmease Oct 12, 2024
6e83863
update hang test to use polars with pycapsule path
jonmmease Oct 12, 2024
50c7163
Add system python of 3.11
jonmmease Oct 12, 2024
751af26
Add more efficient hashing and rechunk for DataFusion
jonmmease Oct 12, 2024
8974fad
fmt
jonmmease Oct 12, 2024
380de1e
use narwhals to remove unused columns.
jonmmease Oct 12, 2024
67fb559
toward removing arrow/pyarrow flag
jonmmease Oct 13, 2024
b9bdf13
Remove arrow-rs pyarrow flag, use pyo3-arrow
jonmmease Oct 13, 2024
68b8324
Rename pyarrow feature flag to `py`
jonmmease Oct 13, 2024
80fd0da
toward using narwhals to process transformed data
jonmmease Oct 14, 2024
b9b65e1
Remove python datasource
jonmmease Oct 14, 2024
57e5efc
Handle dict and non-narwhals pycapsule types
jonmmease Oct 14, 2024
f9f7961
update default extraction to arro3
jonmmease Oct 14, 2024
f27c7f7
fix type checking
jonmmease Oct 14, 2024
7d8b9ca
build py before type checking
jonmmease Oct 14, 2024
47f675f
skip empty fields in window transform
jonmmease Oct 14, 2024
e029456
Try normalize category order
jonmmease Oct 15, 2024
1dad584
lower min Python back to 3.9
jonmmease Oct 15, 2024
9cefb4c
clear wheel build dir first
jonmmease Oct 15, 2024
23efb08
try rename artifacts
jonmmease Oct 15, 2024
56fa42a
cache?
jonmmease Oct 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 32 additions & 23 deletions .github/workflows/build_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-check-rust
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-check-rust
path: |
~/.cargo
target
Expand Down Expand Up @@ -45,7 +45,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-test-rust2
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-test-rust2
path: |
~/.cargo
target
Expand Down Expand Up @@ -74,7 +74,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-wasm
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-wasm
path: |
~/.cargo
target
Expand All @@ -99,11 +99,14 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-wasm
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-wasm
path: |
~/.cargo
target
.pixi
- name: Dev python
run: |
pixi run dev-py
- name: Check format and lint
run: |
pixi run lint-check-py
Expand All @@ -127,6 +130,8 @@ jobs:
uses: Swatinem/rust-cache@81d053bdb0871dcd3f10763c8cc60d0adc41762b # pin@v1
with:
cache-on-failure: True
- name: Clear target/wheels
run: rm -rf target/wheels
- name: Build wheels (Linux)
uses: messense/maturin-action@60d11847b29f81ca5375519a8eb33cc336ba4bfa # [email protected]
with:
Expand All @@ -137,7 +142,7 @@ jobs:
- name: Upload artifacts
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # [email protected]
with:
name: vegafusion-python-wheels
name: python-wheels
path: |
target/wheels/*.tar.gz
target/wheels/*.whl
Expand All @@ -159,6 +164,8 @@ jobs:
- name: Download arm64 toolchain
run: |
rustup target add aarch64-unknown-linux-gnu
- name: Clear target/wheels
run: rm -rf target/wheels
- name: Build arm64 wheels
uses: messense/maturin-action@60d11847b29f81ca5375519a8eb33cc336ba4bfa # [email protected]
with:
Expand All @@ -169,7 +176,7 @@ jobs:
- name: Upload artifacts
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # [email protected]
with:
name: vegafusion-python-wheels
name: python-wheels
path: |
target/wheels/*.tar.gz
target/wheels/*.whl
Expand All @@ -179,13 +186,17 @@ jobs:
steps:
- name: Check out repository code
uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # [email protected]
- name: Setup Python
uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # [email protected]
with:
python-version: '3.11'
- uses: prefix-dev/[email protected]
with:
pixi-version: v0.30.0
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-python-win-64
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-python-win-64
path: |
~/.cargo
target
Expand All @@ -196,7 +207,7 @@ jobs:
- name: Upload artifacts
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # [email protected]
with:
name: vegafusion-python-wheels
name: python-wheels
path: |
target/wheels/*.tar.gz
target/wheels/*.whl
Expand All @@ -212,7 +223,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-python-osx-64
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-python-osx-64
path: |
~/.cargo
target
Expand All @@ -223,7 +234,7 @@ jobs:
- name: Upload artifacts
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # [email protected]
with:
name: vegafusion-python-wheels
name: python-wheels
path: |
target/wheels/*.tar.gz
target/wheels/*.whl
Expand All @@ -239,7 +250,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: osx-arm64-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-python-osx-arm64
key: osx-arm64-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-python-osx-arm64
path: |
~/.cargo
target
Expand All @@ -250,7 +261,7 @@ jobs:
- name: Upload artifacts
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # [email protected]
with:
name: vegafusion-python-wheels
name: python-wheels
path: |
target/wheels/*.tar.gz
target/wheels/*.whl
Expand All @@ -277,15 +288,14 @@ jobs:
- name: Download vegafusion-python wheel
uses: actions/download-artifact@cbed621e49e4c01b044d60f6c80ea4ed6328b281 # pin@v2
with:
name: vegafusion-python-wheels
name: python-wheels
path: target/wheels/
- name: install wheels
working-directory: target/wheels/
run: |
ls -la
python -m pip install vegafusion-*manylinux_2_17_x86_64*.whl


# Optional dependencies
python -m pip install polars-lts-cpu "duckdb>=1.0" "vl-convert-python>=1.0.1rc1" scikit-image "pandas>=2.2" jupytext voila anywidget ipywidgets chromedriver-binary-auto

Expand Down Expand Up @@ -320,15 +330,14 @@ jobs:
- name: Download vegafusion-python wheel
uses: actions/download-artifact@cbed621e49e4c01b044d60f6c80ea4ed6328b281 # pin@v2
with:
name: vegafusion-python-wheels
name: python-wheels
path: target/wheels/
- name: install wheels
working-directory: target/wheels/
run: |
ls -la
python -m pip install vegafusion-*macosx_11_*_arm64.whl


# Optional dependencies
python -m pip install polars "duckdb>=1.0" vl-convert-python "pandas>=2.2"

Expand Down Expand Up @@ -356,7 +365,7 @@ jobs:
- name: Download vegafusion-python wheel
uses: actions/download-artifact@cbed621e49e4c01b044d60f6c80ea4ed6328b281 # pin@v2
with:
name: vegafusion-python-wheels
name: python-wheels
path: target/wheels/
- name: install wheels
working-directory: target/wheels/
Expand All @@ -368,7 +377,7 @@ jobs:
python -m pip install $vegafusion

# Optional dependencies
python -m pip install pyarrow==10.0 altair==5.1.2 "numpy<2" polars[timezone] "duckdb>=1.0" vl-convert-python
python -m pip install "numpy<2" polars[timezone] "duckdb>=1.0" vl-convert-python

# Test dependencies
python -m pip install pytest altair vega-datasets scikit-image
Expand All @@ -388,7 +397,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-server-linux-64
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-server-linux-64
path: |
~/.cargo
target
Expand Down Expand Up @@ -422,7 +431,7 @@ jobs:
# - name: Cache
# uses: actions/cache@v3
# with:
# key: linux-arm64-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-server
# key: linux-arm64-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-server
# path: |
# ~/.cargo
# target
Expand Down Expand Up @@ -497,7 +506,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-server-win-64b
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-server-win-64b
path: |
~/.cargo
target
Expand Down Expand Up @@ -529,7 +538,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-server-osx-64
key: ${{ runner.os }}-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-server-osx-64
path: |
~/.cargo
target
Expand Down Expand Up @@ -560,7 +569,7 @@ jobs:
- name: Cache
uses: actions/cache@v3
with:
key: osx-arm64-${{ hashFiles('pixi.lock', 'Cargo.lock') }}-build-vegafusion-server-osx-arm64
key: osx-arm64-${{ hashFiles('pixi.lock', 'Cargo.lock', 'vegafusion-python/pyproject.toml') }}-build-vegafusion-server-osx-arm64
path: |
~/.cargo
target
Expand Down
Loading
Loading