Enable Cache Reuse for ExtMemQuantileDMatrix Across Sessions

[scottflorence]

Currently, ExtMemQuantileDMatrix creates cache files with names that include the memory address of the DMatrix object (e.g., cache-0x7f8b4c000000.gradient_index.page). This makes it impossible to reuse these cache files across different process sessions, even when working with the exact same data.

Although I understand that this design feature was on purpose, it would be good if we can enable something similar as it would save a large amount of pre-computing the same data if we could reuse the .page files generated, as long as we know they are being generated from the same input DataIter.

[trivialfis]

Thank you for raising the issue, we are planning to add the serialization to the external memory DMatrix.

We will add support for saving the DMatrix into a different binary as the cache is incomplete.(We keep some parts of it in memory for performance reasons, the cache itself is not the complete DMatrix)

[scottflorence]

Thanks for the quick response. Is there any plan to plan to add serialization to any of the QuantileDMatrix variants? I benefit a lot from the compression through quantization, so being able to write it to a file would help massively. There is already a request for the regular QuantileDMatrix (#9545) but I imagine this would also be beneficial for ExtMemQuantileDMatrix also.

[trivialfis]

Yes, the ExtMemQdm is the primary objective.

[scottflorence]

That will be great. Thank you