How to compare two Daft DataFrames in Daft way?

[hongbo-miao]

In pandas, we can use .equals to compare two DataFrames in unit test.

One workaround way is using

daft_df1.collect().to_pandas().equals(daft_df2.collect().to_pandas())

I saw two related open tickets at

• dataframe equality #2590
• Assert Dataframe and schema for testing #3943

What is recommended efficient way to compare two Daft DataFrame? Any Daft way? Thanks! ☺️

[universalmind303]

Hi @hongbo-miao.

Internally, we mostly use .to_pydict() to test equality, and this is fine for smaller datasets, but for larger ones, its not ideal or efficient.

Since our dataframe can be in several states, and potentially contain partitioned data, it's not as trivial as pandas/polars df.equals. We'll look into how we can better implement an equals function that works for our dataframes. In the meantime, I suggest either comparing the plan, schema, or comparing the materialized values via to_pydict.

[colin-ho]

also to_arrow will be more performant than to_pydict as it doesn't need to cast every value to a python object.

[hongbo-miao]

Thanks @universalmind303 and @colin-ho for the info! I think using to_arrow is a good approach for now.
Hopefully, Daft’s equals method will be ready soon.