Description
Wanted to open a discussion about the intended behaviour of SPLIT_TIME_THRESHOLD in conjunction with SPLIT_OUTPUT_FILE.
Context
In my CI setup, the original timings.json file is kept in cypress/e2e/<sub-project>/timings.json, and I am setting the SPLIT_TIME_THRESHOLD to be 0.2.
I am also using SPLIT_OUTPUT_FILE to save the updated timings for each parallel node into their own artifacts directory like so:
artifacts/<sub-project>/split-timings/node-<1,2,3,etc>/timings.json. These are then merged back together by a later merge job using the cypress-split-merge binary (which is working great btw!)
Expected Behaviour (in my head)
The behaviour I was expecting here was that the SPLIT_OUTPUT_FILE would not include updated timings unless the threshold is met. If the threshold was not met, it would either:
- not write out an updated timings.json at all (this might not work well when merging later)
- or it would write to the
node-Xoutput file, but it would be the original times of the test cases and not the updated ones. in this case, when merging the updated timings, Git can realise there is no change to the merged timings file and not bother committing.
Actual Behaviour
Despite the changes not meeting the threshold, it writes the updated timings to the SPLIT_OUTPUT_FILE anyway:
I can see in the code it's because OUTPUT !== the INPUT file, so we write it anyway:
Line 256 in 16f7a78
As a result of the above, later in our merge job the spec timings are being updated no matter what the threshold is.
Question for you
What would be the ideal way for SPLIT_OUTPUT_FILE to respect SPLIT_TIME_THRESHOLD? Would love to help hop on a solution!