Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dask empty DataFrames aren't really empty #287

Open
smcguire-cmu opened this issue Apr 23, 2024 · 0 comments
Open

Dask empty DataFrames aren't really empty #287

smcguire-cmu opened this issue Apr 23, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@smcguire-cmu
Copy link
Contributor

smcguire-cmu commented Apr 23, 2024

In places where we create an empty dask DataFrame, such as in a soft association table or after a filter that doesn't create any partitions, we don't actually make an empty Dask DataFrame. In dask versions < 2024.3.0 without dask expressions dask would make a single partition with the pandas empty df from the meta. In dask >= 2024.3.0, from_delayed with an empty list of inputs raises an error so with #288, we create the empty single partition ourselves. For empty catalogs with no pixels, I think it makes sense to have an empty dask df with no partitions if possible.

@smcguire-cmu smcguire-cmu added the bug Something isn't working label Apr 23, 2024
@smcguire-cmu smcguire-cmu changed the title Dask empty DataFrames aren' Dask empty DataFrames aren't really empty Apr 23, 2024
@nevencaplar nevencaplar moved this to Todo in HATS / LSDB Apr 25, 2024
@delucchi-cmu delucchi-cmu removed the status in HATS / LSDB May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: No status
Development

No branches or pull requests

1 participant