Conversation
|
@LHerviou How do you feel about these changes? I am mostly wondering about how to organize the files, in the sense that it might be reasonable to simply store all Fs and all Rs each in a single file. |
Codecov ReportAttention: Patch coverage is
🚀 New features to boost your workflow:
|
|
@lkdvos Thanks for such a quick implementation. I think a single file should be fine: read/write access should be much faster. It might be worth thinking if having F_N_abc and R_N_ab is not a better alternative -- though it does increase significantly the number of files. I am indeed worried about the corruption on clusters where you can have instabilities. One or two quick checks/questions for parallel runs:
|
|
Just to be clear: the current implementation uses separate files Whenever an F or R or CGC is called, the callstack goes:
where computing F and R asks for the CGC, which will again go through the same process. I'll add a switch for turning disk storage on and off as well, so you can just select which ones you want to store persistently? |
This PR updates the Fsymbols and Rsymbols to be stored on disk as well.
The main motivation is that typically, the CGCs take up way more space than the actual F and Rsymbols, so it may be reasonable to have a single machine that precomputes a lot of them and then not bother with copying the (possible terrabytes of) CGCs.
For reference, I currently have: