BUG: Inconsistent BOM handling in pd.read_csv with encoding='utf-8'

[MikiPAUL]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import io

# CSV with UTF‑8 BOM
bytes_data = b"\xef\xbb\xbfName,Age\nJohn,25"
bio = io.BytesIO(bytes_data)


df_c = pd.read_csv(bio, encoding="utf-8")
print(repr(df_c.columns[0]))  # 'Name'

bio.seek(0)
# but different behaviour in text wrapper
stream = io.TextIOWrapper(bio, encoding='utf-8')

print(repr(stream.read())) # '\ufeffName,Age\nJohn,25'

Issue Description

When reading a UTF-8 encoded CSV that contains a BOM, specifying encoding='utf-8' pandas strips the BOM from the first column header and textwrapper leaves it as an invisible \ufeff character. The outcome depends on the parsing, but this behaviour is undocumented and inconsistent.

Expected Behavior

I expected encoding='utf-8' either to always keep the BOM (requiring utf-8-sig to remove it) or to always strip it, regardless of parser. At minimum, the documentation should clarify how BOMs are handled across engines when using encoding='utf-8'.

Installed Versions

Details

Replace this line with the output of pd.show_versions()

[rhshadrach]

Thanks for the report!

pandas strips the BOM from the first column header and textwrapper leaves it as an invisible \ufeff character. The outcome depends on the parsing, but this behaviour is undocumented and inconsistent.

I'm not well versed in the details of BOM parsing, but I fail to see why pandas' CSV parser and Python's TextIOWrapper must be consistent when it comes to handling of BOM. Can you elaborate here?

[MikiPAUL]

@rhshadrach Shouldn't the BOM parsing happens only when the encoding is utf-8-sig?

https://docs.python.org/3/library/codecs.html#module-encodings.utf_8_sig
https://tedboy.github.io/python_stdlib/_modules/encodings/utf_8_sig.html

[rhshadrach]

Here is something I can call a bug.

df = pd.DataFrame(
    {
        "ABC": [1, 2, 3],
    }
)
df.to_csv("test.csv", index=False, encoding="iso-8859-1")
print(pd.read_csv("test.csv", encoding="utf-8"))
#    ABC
# 0    1
# 1    2
# 2    3

Agreed this shouldn't happen, PRs to fix are welcome!

[sanrishi]

@rhshadrach

I looked into this. It seems the C-tokenizer is aggressively sniffing and skipping BOM bytes before the codec layer even gets them.

To fix this without breaking backward compatibility (like for Excel files), I propose a tri-state logic in the wrapper that passes a flag down to the C-engine:

1. encoding=None (Default): Strip BOM (Keeps current "helpful" behavior for existing users).

2. encoding='utf-8-sig': Strip BOM (Explicit user intent).

3. encoding='utf-8': Preserve BOM (Fixes the bug, aligns with Python standard).

Basically, this ensures read_csv("excel.csv") keeps working cleanly, but read_csv("...", encoding="utf-8") strictly respects the bytes.

Plan is to plumb a strip_bom flag from CParserWrapper down to tokenizer.c. Does this sound right to you?