Use same defaults as `pandas.DataFrame.to_csv` in `kedro_datasets.pandas.csv_dataset.CSVDataset`.

[galenseilisnh]

Description

I ran a pipeline and saved the results only to find out that the index, which contained required information, was missing!

Context

Users are expecting the pandas.CSVDataset to be a shallow wrapper around pandas.DataFrame.to_csv. It essentially is, but along with that shallow wrapper expectation is the expectation that the defaults will be the same. I found it surprising that the defaults were not preserved. It isn't a huge deal, but it becomes a "gotcha" for anyone onboarding Kedro + pandas CSV.

Possible Alternatives

Naturally, users 'can' use the code in the current state. This isn't a huge problem. Just a footgun for new users.

[rashidakanchwala]

Thanks for flagging this — good point. Ideally, we should match Pandas, but changing this now would be a breaking change. I think the best path is to update the docstring and docs to make this clearer for users.

@merelcht what are your thoughts on this issue?

[galenseilisnh]

I think the best path is to update the docstring and docs to make this clearer for users.

That's a practical solution for the short term. In the long term I'd prefer this change be made in a major release.

Kedro users, including myself, will often accept breaking changes like this if they are communicated in advance and don't require substantial work to fix from the user end of things. The user code that would have to change here could be done for a large number of projects by a script, so from my point of view the changeover costs would be minor even for a large number of projects. That's my opinion as a user of Kedro, and a developer of some internal tools in Python, anyway. Hopefully it is useful to hear that.

[merelcht]

If the changes only need to happen in kedro-dataset, breaking it is no big deal. The next release is breaking anyway.

[merelcht]

@galenseilisnh , just to clarify is this specifically for the value "index": False ?

[galenseilisnh]

@galenseilisnh , just to clarify is this specifically for the value "index": False ?

@merelcht Good question. Yes, that is the parameter and argument implicit in the original post.