Spaces at the end of CSV files

[MBuchalik]

I tried loading a CSV file (using the Adapter) with a structure like this:

column1,column2,column3
first,second,third

There are a few newlines after the second row.

Expected result

I would expect to only get two rows in the result like so:

[
  ["column1", "column2", "column3"],
  ["first", "second", "third"]
]

Actual result

I am getting multiple "empty" rows at the end:

[
  ["column1","column2","column3"],
  ["first","second","third"],
  [""],
  [""],
  [""]
]

Here is a sample CSV file for you to try it out:
spaces.zip

A similar issue happens if there is an empty line between the rows like here: spaces2.zip. But I am not sure if we should clean that up or not. (Intuitively, I would say "yes")

[sonallux]

As CSV is a very loose standard with many variants, I would argue removing and keeping the blank lines are both valid options. As there is an option CsvParser.Feature.SKIP_BLANK_LINES in our CSV parser, I would add a new parameter to our CSV format to let the user decide what he wants.

[MBuchalik]

Do you have a sample data set for me where the blank lines actually have a "meaning"? I am just trying to figure out in which cases you would actually need the empty rows.

[sonallux]

Do you have a sample data set for me where the blank lines actually have a "meaning"?

Yes, I can think of a case where you are doing exactly 50 measurements with a defined order. Some measurements might not lead to value (e.g. measuring device defect), which is represented as a black line. When now removing the black lines, the information about the missing measurement is lost.

[MBuchalik]

Thanks for the example 👍

We should probably talk about this in the next meeting before implementing something. Reason: We will create a feature for data validation ("column 2 is a number > 5") in the near future. Allowing empty lines could make this feature harder to implement/understand.

Altenatively, we could add a "row number" column. Then, we could always skip the empty lines and users would still be able to figure out that something got skipped. But that could also introduce other issues...

[mathiaszinnen]

The issue is has been posted here.
We should try CsvGenerator.Feature.STRICT_CHECK_FOR_QUOTING first.