BUG: combine_first casts dtypes, pandas does not

[Liquidmasl]

Modin version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest released version of Modin.

• I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example

Pandas

import pandas as pd
ddf1 = pd.DataFrame({"a":[1,2,3], "b":[4,5,6]})
ddf2 = pd.DataFrame({"a":[10,20,30], "c":[4,5,6]})
ddf1.combine_first(ddf2)
#     a  b  c
#  0  1  4  4
#  1  2  5  5
#  2  3  6  6

ddf1.combine_first(ddf2).dtypes
#  a    int64
#  b    int64
#  c    int64
#  dtype: object


Modin:

import modin.pandas as mpd
df1 = mpd.DataFrame({"a":[1,2,3], "b":[4,5,6]})
df2 = mpd.DataFrame({"a":[10,20,30], "c":[4,5,6]})

df1.combine_first(df2)
#      a  b    c
#  0   1  4  4.0
#  1   2  5  5.0
#  2   3  6  6.0

df1.combine_first(df2).dtypes
#  a       int64
#  b       int64
#  c    float64
#  dtype: object

Issue Description

int type should not be converted to float.

This is especially problematic with huge datasets, this explodes the amout of ram needed. And it a very hidden and weird issue.

Might be related to #5959, but that has not gotten any attention in a few years.

This is currently a real issue in our codebase.

Expected Behavior

Behave like pandas, column should stay int

Error Logs

No response

Installed Versions

INSTALLED VERSIONS

commit : 14680b3
python : 3.12.9
python-bits : 64
OS : Linux
OS-release : 6.8.0-52-generic
Version : #53~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jan 15 19:18:46 UTC 2
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
Modin dependencies

modin : 0.34.1
ray : 2.48.0
dask : None
distributed : None
pandas dependencies

pandas : 2.2.3
numpy : 2.2.6
pytz : 2025.2
dateutil : 2.9.0.post0
pip : None
Cython : None
sphinx : 8.2.3
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.3.2
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : 3.10.3
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 17.0.0
pyreadstat : None
pytest : 8.4.1
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.16.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None

[sfc-gh-joshi]

Thanks for the report. The issue you linked seems to actually have been closed a while ago, so the root cause is something different.

I did some digging and it looks like this occurs because of how we perform type inference in binary operations:

modin/modin/core/dataframe/algebra/binary.py

Lines 80 to 87 in e3724b6

	if isinstance(second, type(first)):
	dtypes_second = second.dtypes.to_dict()
	columns_first = set(first.columns)
	columns_second = set(second.columns)
	common_columns = columns_first.intersection(columns_second)
	# Here we want to XOR the sets in order to find the columns that do not
	# belong to the intersection, these will be NaN columns in the result
	mismatch_columns = columns_first ^ columns_second

modin/modin/core/dataframe/algebra/binary.py

Lines 164 to 172 in e3724b6

	dtypes: pandas.Series = pandas.concat(
	[
	dtypes,
	pandas.Series(
	[nan_dtype] * (len(mismatch_columns)),
	index=mismatch_columns,
	),
	]
	)

When a column isn't present in the left frame, we pick the dtype of np.nan, which is float64, and since "c" isn't present in the left frame of this combine operation, this nan dtype path is run. At a glance it looks like combine and combine_first are the only operations that use this code path, but I'll need to do some more digging to check that other join operations wouldn't break if we switched the behavior to use dtypes of existing columns instead.

[Liquidmasl]

that was a superfast responst, thank you for looking into it!

[sfc-gh-joshi]

Did some more investigation, fixing this properly will be more difficult than I initially thought. It might be a while before I have time to return to this, so I'm leaving my findings here in case someone else is able to take this task on.

Changing the inferred result dtypes of the top-level object computed by Binary.register isn't sufficient, because this is independent of the actual types that are computed under the hood by Modin's partitioning logic. For most binary operations involving a join, if a column is missing from one frame, setting its values to np.NaN and the dtype to float64 is expected:

>>> df1 = pd.DataFrame({"a": [1]})
>>> df2 = pd.DataFrame({"a": [3], "b": [2]})
>>> df1 + df2  # Same behavior in pandas and modin.pandas
   a   b
0  4 NaN
>>> (df1 + df2).dtypes
a      int64
b    float64
dtype: object

It's worth noting that in pandas, combine_first will match the dtype of a column in the left frame, even if its values are all null.

>>> import pandas as pd
>>> pd.DataFrame({"a": [np.nan]}).combine_first(pd.DataFrame({"a": [1]}))
     a
0  1.0

Note that the result is a float, even when the operation's non-null RHS is integral. This is the root cause of the modin bug: when we perform an n-ary operation (query compiler method), we always compute the join result's index first, and fill in columns of None/nan as needed in each argument. When we apply the pandas combine_first operation on each partition, it then follows the behavior of the pandas code snippet I pasted above, and will accept the dtype of the left column even if its values are all null.

A band-aid fix for this would be to astype each column, but if the original issue is causing problems with RAM usage I suspect such a fix would only make things worse. If we want to be as memory-efficient as possible, we should find a way to avoid creating null placeholder columns specifically for combine operations (or just use an appropriately-dtyped null value instead when reindexing), but it's been a while since I've had to dig into partitioning code and doing this correctly could be tricky.