Order of rows in crosscheck metrics files are non-deterministic

[kachulis]

The order of rows output by CrosscheckFingerprints seems to change from run to run on the same input data. Not sure if this is new behavior or has always been the case.

This isn't a massive problem, but would be nice for the order to always be the same.

[lbergelson]

As always I suspect HashMap/Set

[serge2016]

Is it possible to fix it?

[droazen]

Reopening, as this was not completely fixed by #1892

[droazen]

Remaining TODOs (courtesy of @lbergelson ):

I looked at this a bit more, it seems like there is instability built in at the start since FingerPrintChecker.fingerprintFiles() builds an unsorted map from the output of a parallel and unpredictable process. I think it's going to require a more complicated change than just changing the other sets to linked sets. Either change the inputs to return their results in order, or sort the maps according to some criteria.

I think we either need to change it so the input maps are built as the result of Futures instead of being added to the concurrent map as the are built. The errors are actually handled this way so it wouldn't be that hard to change.

The other option would be to pick an arbitrary sort order and sort the input maps after they're built and then maintain that order.