Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement virtual arrays #3364

Draft
wants to merge 30 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
8407680
first try to implement virtual arrays at (least close to) the buffer …
pfackeldey Dec 21, 2024
938b883
lint
pfackeldey Dec 21, 2024
eccb9a8
prepare virtual arrays for Iason (now really at the buffer level)
pfackeldey Dec 23, 2024
960f7c3
fix .T for virtual arrays
pfackeldey Dec 23, 2024
ecb883b
minimal changes to get an example of a listoffsetarray with virtual b…
ikrommyd Jan 21, 2025
40824c7
what if we start like this?
ikrommyd Jan 21, 2025
7b2809c
add repr
ikrommyd Jan 21, 2025
85d8a83
restore _data <-> data to original
ikrommyd Jan 21, 2025
3b5ac6a
an attempt at implementing some methods
ikrommyd Jan 21, 2025
6b8cbc4
style: pre-commit fixes
pre-commit-ci[bot] Jan 21, 2025
d0e4947
tiny changes
ikrommyd Jan 21, 2025
a91dad0
add ones_like and full_like to play around a bit
ikrommyd Jan 21, 2025
40841a2
no need to implement ufuncs separately. They are covered by apply_ufunc
ikrommyd Jan 22, 2025
72babd0
add materialization in array_module.py
ikrommyd Jan 22, 2025
95e9e3c
same repr for placeholders
ikrommyd Jan 22, 2025
6a25b1b
adhere to ArrayLike interface
ikrommyd Jan 22, 2025
c1ff9ea
add TODO comment
ikrommyd Jan 22, 2025
166fcb0
try kernels
ikrommyd Jan 23, 2025
c5020d0
never create new virtual arrays for now, let them materialize
ikrommyd Jan 23, 2025
7a8a074
ruff fix
ikrommyd Jan 23, 2025
b2bb788
bring back creation of virtual arrays with a local cache
ikrommyd Jan 25, 2025
6e41bda
forgot this
ikrommyd Jan 25, 2025
a84d84e
Merge branch 'main' into virtual-arrays
ikrommyd Jan 25, 2025
4137bf9
maybe rapidjson submodule fix
ikrommyd Jan 25, 2025
c773329
adjust _get_item_at_placeholder for virtual arrays as well
ikrommyd Jan 25, 2025
391fe17
new _get_item_at_virtual function and different VirtualValue vs Place…
ikrommyd Jan 26, 2025
80f40ae
remove some comments and docstrings that we decided to use in a diffe…
ikrommyd Jan 26, 2025
93db3a4
let iterators of virtual buffers materialize
ikrommyd Jan 26, 2025
4c36c7d
inherit from NDArrayOperatorsMixin as well
ikrommyd Jan 27, 2025
7c59dba
Merge branch 'main' into virtual-arrays
ikrommyd Jan 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/awkward/_kernels.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from awkward._nplikes.numpy import Numpy
from awkward._nplikes.numpy_like import NumpyMetadata
from awkward._nplikes.typetracer import try_touch_data
from awkward._nplikes.virtual import materialize_if_virtual
from awkward._typing import Protocol, TypeAlias

KernelKeyType: TypeAlias = tuple # Tuple[str, Unpack[Tuple[metadata.dtype, ...]]]
Expand Down Expand Up @@ -88,6 +89,8 @@ def _cast(cls, x, t):
def __call__(self, *args) -> None:
assert len(args) == len(self._impl.argtypes)

args = materialize_if_virtual(*args)

return self._impl(
*(self._cast(x, t) for x, t in zip(args, self._impl.argtypes))
)
Expand All @@ -97,6 +100,8 @@ class JaxKernel(NumpyKernel):
def __call__(self, *args) -> None:
assert len(args) == len(self._impl.argtypes)

args = materialize_if_virtual(*args)

if not any(Jax.is_tracer_type(type(arg)) for arg in args):
return super().__call__(*args)

Expand Down Expand Up @@ -138,6 +143,8 @@ def _cast(self, x, type_):
def __call__(self, *args) -> None:
import awkward._connect.cuda as ak_cuda

args = materialize_if_virtual(*args)

cupy = ak_cuda.import_cupy("Awkward Arrays with CUDA")
maxlength = self.max_length(args)
grid, blocks = self.calc_grid(maxlength), self.calc_blocks(maxlength)
Expand Down
1 change: 1 addition & 0 deletions src/awkward/_nplikes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import awkward._nplikes.jax
import awkward._nplikes.numpy
import awkward._nplikes.typetracer
import awkward._nplikes.virtual
from awkward._nplikes.dispatch import nplike_of_obj
from awkward._typing import TYPE_CHECKING

Expand Down
Loading