Rationalise CSV output

[ns-rse]

Currently we have multiple CSV files output and duplication of statistics across these.

We should rationalise this and minimise the output to images and grains with as little duplication.

Broad overview

As we move towards refactoring with a TopoStats object and a number sub-classes where information is held and these should form the basis of the output. Having functions that pull out the relevant information from these classes and and write to CSV will be a sensible way to achieve this.

[tobyallwood]

I've had a look at combining all_statistics.csv and all_molecule_statistics.csv (as they have the most similar structure) by manually combining the first few grains/ molecules and adding a new column specifying each row's type, and have come up with this:

all_statistics example merge.csv

While this works, I think it's much harder to quickly skim through and read than the original separate csvs. What do you think though? A solution could be using an xlsx file instead so we could colour-code the rows by row_type (for example) and have multiple sheets in one file but I'm not sure why that could be a bad idea. Let me know though

[ns-rse]

Thanks @tobyallwood

I think this has potential to get confusing as the structure at present doesn't show which grain a given molecule relates to, its only inferred from the order of the rows as molecules follow grains and if that changed (quite easy to do) then you no longer know what grain a given molecule relates to.

There are two approaches to how we might structure this...

• Long format
• Wide format

There are pros/cons to each, personally I prefer long format, but the drawback here is that it would result in duplicated statistics for grain level statistics and anyone hoping to just load and plot such a file would have to deduplicate it first.

Long format

NB I've deliberately removed _number from column names here, not a fan of including that in variable names as it adds nothing informative.

image	grain	class	subgrain	molecule	centre_x	center_y	radius_min	...other grain statistics...	circular	topology	topology_flip	processing	contour_length	end_to_end_distance
img1	0	1	1	0	1.2	1.8	9	...	True	0_1	0_1	nodestats	0.0000868	0
img1	0	1	1	1	1.2	1.8	9	...	False	linear		nodestats	0.0000468	0.000005
img1	0	1	1	2	1.2	1.8	9	...	True	0_2	0_2	topostats	0.0000315	0
img1	1	2	1	0	4.6	5.3	9	...	False	linear		nodestats	0.0000123	0.000003
img1	1	2	1	1	4.6	5.3	9	...	False	linear		nodestats	0.0000456	0.000008
img12	0	1	1	0	1.2	1.8	9	...	True	0_1	0_1	nodestats	0.0000111	0
img12	0	1	1	1	1.2	1.8	9	...	True	1_0	1_0	nodestats	0.0000222	0
img12	0	1	1	2	1.2	1.8	9	...	True	1_2	1_2	nodestats	0.0000333	0
img12	1	2	1	0	4.6	5.3	9	...	False	linear		topostats	0.0000444	0.000001
img12	1	2	1	1	4.6	5.3	9	...	False	linear		topostats	0.0000555	0.000002
img12	1	2	1	2	4.6	5.3	9	...	False	linear		topostats	0.0000666	0.000003

Because molecule is a subset of the higher level grouping image / grain / class / subgrain the attributes of this higher level grouping are duplicated so that we can keep track of which each molecule belongs to.

Summary statistics of molecule attributes here is straight forward as they are no duplicate rows/data points. However, if users wanted to take the summary statistics of one radius_min they should really subset the image / grain / class / subgrain and radius_min and deduplicate it before calculating any statistics. This is an extra step for users with varying experience in data analysis be that in Python (Pandas/Polars), R or spreadsheets (I don't even know if it is possible to deduplicate data in spreadsheets?).

Wide Format

In wide format the columns for statistics for each molecule are given identifiers to say which molecule they pertain to.

image	grain	class	subgrain	molecule	centre_x	center_y	radius_min	...other grain statistics...	circular_0	topology_0	topology_flip_0	processing_0	contour_length_0	end_to_end_distance_0	circular_1	topology_1	topology_flip_1	processing_1	contour_length_1	end_to_end_distance_1	circular_2	topology_2	topology_flip_2	processing_2	contour_length_2	end_to_end_distance_2
img1	0	1	1	0	1.2	1.8	9	...	True	0_1	0_1	nodestats	0.0000868	0	False	linear		nodestats	0.0000468	0.000005	True	0_2	0_2	topostats	0.0000315	0
img1	1	2	1	0	4.6	5.3	9	...	False	linear		nodestats	0.0000123	0.000003	False	linear		nodestats	0.0000456	0.000008	NA	NA	NA	NA	NA	NA
img12	0	1	1	0	1.2	1.8	9	...	True	0_1	0_1	nodestats	0.0000111	0	True	1_0	1_0	nodestats	0.0000222	0	True	1_2	1_2	nodestats	0.0000333	0
img12	1	2	1	0	4.6	5.3	9	...	False	linear		topostats	0.0000444	0.000001	False	linear		topostats	0.0000555	0.000002	False	linear		topostats	0.0000666	0.000003

There is no need to subset and deduplicate the grain level statistics here but wide format introduces a challenge if a user wanted to summarise the molecule level statistics as the data would need reshaping to long format in order to get summary statistics of say contour_length as in wide format they are stored in multiple columns (three in this instance).

This also has the potential to introduce an unknown number of additional columns contingent on the number of molecules identified within a given grain.

Why not Excel?

A solution could be using an xlsx file instead so we could colour-code the rows by row_type (for example) and have
multiple sheets in one file but I'm not sure why that could be a bad idea.

There are many reasons why proprietary spreadsheets are a bad idea both principled (ties people into expensive software when free open source alternatives exist) and practical (you have no track record of the changes that are made to data in spreadsheets and its very easy to make mistakes that can have dramatic real-world impacts, e.g. the exclusion of data in the Growth in the Time of Debt paper which underpinned the decision to introduce austerity measures in the UK but see also¹²).

• It would be additional fickle work to add colour formatting.
• Not everyone uses spreadsheets and some export to CSV anyway.
• Make a mistake in a spreadsheet and you have no record of what changes you might have made and no way of repeating the workflow (Python/R scripts or Notebooks avoid this and I advocate for learning how to use these alternative tools).
• If users want to view CSV in a spreadsheet they can easily import them.

I always wonder why people want to look at tables of raw numbers anyway? I for one am unable to look at a spreadsheet and draw out anything other than "_these are numbers_" without recourse to summarising and visualising the data. Summary statistics (e.g. min/max (range), mean, standard deviation, median, inter-quartile range) are far more important to tell you something about the data and a nice visualisation will show the spread of the data and how variables relate to each other.

Summary

If these are to be combined I would advocate for long format, its easier to work with (filtering and subsetting) than reshaping data which takes some getting used to.

Either way of combining the grain and molecule statistics introduces the need for users to have the knowledge and skill t perform some data manipulations. Personally I think all scientific researchers should have these basic data analysis skills to get simple summary statistics as at some point they have to do some statistics and plotting. I don't think it is unreasonable to document how to do this so that users can learn how to do so (writing helper functions
doesn't give them useful skills to take forward in their career).

That said I wonder if we are introducing unnecessary complications for end users by combining files here 🤔

Footnotes

1. Spreadsheet errors can have disastrous consequences – yet we keep making the same mistakes and the cited study which found 90% of spreadsheets containing more than 150 rows have at least one major mistake ↩

2. Excel errors: the UK government has an embarrassingly long history of spreadsheet horror stories ↩

[tobyallwood]

@ns-rse yeah I think I agree with questioning now necessary combining the files would actually be, in an ideal world I understand it would be nice to just have a single CSV outputted after running topostats but every solution I can think of ends up complicating things for the user - either when viewing the CSV in an editor or getting statistics through a dataframe the CSV has been loaded into.

I'm gonna make a short presentation showing this along with what you've said here and hopefully can present it next week to the group