cluster_pairwise_predictions_at_threshold() failing when linking two 140M records

[reyvasquez-vh]

Previously, I had issues with linking two 35M records. See #1017
That works now. Thanks!

I am now trying to link two 140M records, and the code fails at linker.cluster_pairwise_predictions_at_threshold()
The execution printouts a lot of "No more replicas available"

And also:
A lot of Failed to send RPC RPC 5462906116369655171 to /<ApplicationMaster IP>:35100: error

[reyvasquez-vh]

The problem here was with storage for checkpoints. Pointing checkpoint to S3 bucket instead of using node spaces fixed the issue.

[RobinL]

thanks - appreciate you posting the solution